2024-04-16T05:16:10.007 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-16T05:16:10.007 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/vshankar-2024-04-16_05:08:59-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657906 --name vshankar-2024-04-16_05:08:59-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi --verbose --owner scheduled_vshankar@teuthology --description fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} -- /home/teuthworker/archive/vshankar-2024-04-16_05:08:59-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657906/orig.config.yaml 2024-04-16T05:16:10.093 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T05:16:10.215 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-04-16_05:08:59-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657906 branch: wip-vshankar-testing-20240412.042930 description: fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: vshankar@redhat.com first_in_suite: false job_id: '7657906' kernel: flavor: default kdb: true sha1: 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/wip-64502-quincy-3 ' name: vshankar-2024-04-16_05:08:59-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20240412.042930 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-64502-quincy-3 sha1: 96e30c031480d623ee32b6336cfd0a901a764a51 owner: scheduled_vshankar@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 5451 sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 sleep_before_teardown: 0 suite: fs:upgrade:mds_upgrade_sequence suite_branch: wip-64502-quincy-3 suite_path: /home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa suite_relpath: qa suite_repo: https://github.com/vshankar/ceph.git suite_sha1: 96e30c031480d623ee32b6336cfd0a901a764a51 targets: smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAUKOJJtb1jCnhSbBLTrDEMvl7nkN2zXTZYZ0tHH/3dXhQSwBrKySq1C+3BTRBW6QVgU06p2SdGdxkGp4SPmO50= smithi122.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFCcWLoCaHgr/ZCbWUO1jcAcLvGuaJdjPflJ4wydBj+pSal71/S2LrmF3xIHN+SVzZLGynxLMV2zqpzIVfr5bHI= tasks: - install: branch: wip-64502-quincy-3 exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /cephfs/home/vshankar/src/github.com_vshankar_ceph_wip-64502-quincy-3/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-16_05:08:59 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-16T05:16:10.215 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa; will attempt to use it 2024-04-16T05:16:10.216 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa/tasks 2024-04-16T05:16:10.216 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-16T05:16:10.226 INFO:teuthology.task.internal:Checking packages... 2024-04-16T05:16:10.329 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '218a691417ef57c5d22a8d5cf3c5f41f7b542838' 2024-04-16T05:16:10.329 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T05:16:10.329 INFO:teuthology.packaging:ref: None 2024-04-16T05:16:10.330 INFO:teuthology.packaging:tag: None 2024-04-16T05:16:10.330 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240412.042930 2024-04-16T05:16:10.330 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T05:16:10.330 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20240412.042930 2024-04-16T05:16:10.478 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2884.g218a6914 2024-04-16T05:16:10.479 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-16T05:16:10.489 INFO:teuthology.task.internal:no buildpackages task found 2024-04-16T05:16:10.489 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-16T05:16:10.495 INFO:teuthology.task.internal:Saving configuration 2024-04-16T05:16:10.531 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-16T05:16:10.541 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T05:16:10.574 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-16_05:08:59-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657906', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 05:10:01.021081', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAUKOJJtb1jCnhSbBLTrDEMvl7nkN2zXTZYZ0tHH/3dXhQSwBrKySq1C+3BTRBW6QVgU06p2SdGdxkGp4SPmO50='} 2024-04-16T05:16:10.596 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-16_05:08:59-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657906', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 05:10:01.022669', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFCcWLoCaHgr/ZCbWUO1jcAcLvGuaJdjPflJ4wydBj+pSal71/S2LrmF3xIHN+SVzZLGynxLMV2zqpzIVfr5bHI='} 2024-04-16T05:16:10.597 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-16T05:16:10.603 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-04-16T05:16:10.604 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-04-16T05:16:10.604 INFO:teuthology.run_tasks:Running task console_log... 2024-04-16T05:16:10.749 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5eb99e30d0>, signals=[15]) 2024-04-16T05:16:10.750 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-16T05:16:10.757 INFO:teuthology.task.internal:Opening connections... 2024-04-16T05:16:10.757 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2024-04-16T05:16:10.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:16:10.886 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2024-04-16T05:16:10.887 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:16:11.036 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-16T05:16:11.043 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2024-04-16T05:16:11.062 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2024-04-16T05:16:11.063 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2024-04-16T05:16:11.121 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Stream" 2024-04-16T05:16:11.121 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="9" 2024-04-16T05:16:11.121 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos" 2024-04-16T05:16:11.121 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora" 2024-04-16T05:16:11.121 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="9" 2024-04-16T05:16:11.121 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el9" 2024-04-16T05:16:11.121 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T05:16:11.121 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2024-04-16T05:16:11.121 INFO:teuthology.orchestra.run.smithi027.stdout:LOGO="fedora-logo-icon" 2024-04-16T05:16:11.122 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T05:16:11.122 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/" 2024-04-16T05:16:11.122 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T05:16:11.122 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T05:16:11.122 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T05:16:11.122 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2024-04-16T05:16:11.154 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2024-04-16T05:16:11.172 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2024-04-16T05:16:11.173 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2024-04-16T05:16:11.230 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="CentOS Stream" 2024-04-16T05:16:11.230 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="9" 2024-04-16T05:16:11.230 INFO:teuthology.orchestra.run.smithi122.stdout:ID="centos" 2024-04-16T05:16:11.230 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="rhel fedora" 2024-04-16T05:16:11.230 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="9" 2024-04-16T05:16:11.230 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el9" 2024-04-16T05:16:11.230 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T05:16:11.230 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2024-04-16T05:16:11.230 INFO:teuthology.orchestra.run.smithi122.stdout:LOGO="fedora-logo-icon" 2024-04-16T05:16:11.230 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T05:16:11.231 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://centos.org/" 2024-04-16T05:16:11.231 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T05:16:11.231 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T05:16:11.231 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T05:16:11.231 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2024-04-16T05:16:11.252 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-16T05:16:11.264 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-16T05:16:11.270 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-16T05:16:11.271 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2024-04-16T05:16:11.274 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2024-04-16T05:16:11.290 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-16T05:16:11.304 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-16T05:16:11.304 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2024-04-16T05:16:11.336 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2024-04-16T05:16:11.429 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-16T05:16:11.488 INFO:teuthology.run_tasks:Running task kernel... 2024-04-16T05:16:11.504 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8'} 2024-04-16T05:16:11.504 INFO:teuthology.task.kernel:config {'host.a': {'flavor': 'default', 'kdb': True, 'sha1': '3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8'}, 'host.b': {'flavor': 'default', 'kdb': True, 'sha1': '3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8'}}, timeout 300 2024-04-16T05:16:11.504 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T05:16:11.505 DEBUG:teuthology.orchestra.run.smithi122:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T05:16:11.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T05:16:11.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T05:16:11.526 DEBUG:teuthology.task.kernel:sha1 for host.a is 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T05:16:11.526 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8"... 2024-04-16T05:16:11.528 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-04-16T05:16:11.528 DEBUG:teuthology.task.kernel:sha1 for host.b is 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T05:16:11.528 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8"... 2024-04-16T05:16:11.528 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-04-16T05:16:11.583 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-361.el9.x86_64 2024-04-16T05:16:11.583 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T05:16:11.583 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-16T05:16:11.583 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T05:16:11.587 INFO:teuthology.orchestra.run.smithi027.stdout:5.14.0-361.el9.x86_64 2024-04-16T05:16:11.587 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T05:16:11.587 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-16T05:16:11.587 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T05:16:11.735 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum upgrade -y linux-firmware 2024-04-16T05:16:11.750 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum upgrade -y linux-firmware 2024-04-16T05:16:13.778 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-04-16T05:16:13.785 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-04-16T05:16:15.195 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2024-04-16T05:16:15.425 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-04-16T05:16:16.283 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-16T05:16:16.285 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-16T05:16:18.337 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-16T05:16:22.657 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 9.6 MB/s | 6.2 MB 00:00 2024-04-16T05:16:24.396 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-04-16T05:16:25.723 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-04-16T05:16:30.949 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-04-16T05:16:33.538 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:16:33.546 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:16:33.546 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-04-16T05:16:33.546 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:16:33.546 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2024-04-16T05:16:33.546 INFO:teuthology.orchestra.run.smithi027.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-04-16T05:16:33.546 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:16:33.546 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-16T05:16:33.547 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:16:33.547 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 1 Package 2024-04-16T05:16:33.547 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:16:33.547 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 375 M 2024-04-16T05:16:33.547 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-16T05:16:39.065 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 892 kB/s | 19 MB 00:21 2024-04-16T05:16:40.591 INFO:teuthology.orchestra.run.smithi027.stdout:linux-firmware-20240219-143.el9.noarch.rpm 53 MB/s | 375 MB 00:07 2024-04-16T05:16:40.593 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:16:40.593 INFO:teuthology.orchestra.run.smithi027.stdout:Total 53 MB/s | 375 MB 00:07 2024-04-16T05:16:40.593 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-16T05:16:40.697 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-16T05:16:40.698 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-16T05:16:43.186 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-16T05:16:43.186 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-16T05:16:43.249 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 6.2 MB 00:00 2024-04-16T05:16:44.902 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-16T05:16:45.960 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-04-16T05:16:49.805 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-16T05:16:50.030 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-16T05:16:50.128 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T05:16:51.127 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-04-16T05:16:51.505 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T05:16:51.505 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-16T05:16:51.954 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T05:16:51.954 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:16:51.954 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2024-04-16T05:16:51.954 INFO:teuthology.orchestra.run.smithi027.stdout: linux-firmware-20240219-143.el9.noarch 2024-04-16T05:16:51.954 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:16:51.954 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:16:52.261 INFO:teuthology.task.kernel:Downloading kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on host.a... 2024-04-16T05:16:52.261 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T05:16:52.501 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ 2024-04-16T05:16:52.501 DEBUG:teuthology.orchestra.run.smithi027:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-16T05:16:52.504 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi027.front.sepia.ceph.com to complete... 2024-04-16T05:16:53.624 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:16:53.638 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:16:53.638 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-16T05:16:53.638 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:16:53.638 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrading: 2024-04-16T05:16:53.638 INFO:teuthology.orchestra.run.smithi122.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-04-16T05:16:53.639 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:16:53.639 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-16T05:16:53.639 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:16:53.639 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrade 1 Package 2024-04-16T05:16:53.639 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:16:53.639 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 375 M 2024-04-16T05:16:53.639 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-16T05:17:03.903 INFO:teuthology.orchestra.run.smithi122.stdout:linux-firmware-20240219-143.el9.noarch.rpm 37 MB/s | 375 MB 00:10 2024-04-16T05:17:03.904 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:17:03.904 INFO:teuthology.orchestra.run.smithi122.stdout:Total 37 MB/s | 375 MB 00:10 2024-04-16T05:17:03.905 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-16T05:17:03.987 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-16T05:17:03.987 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-16T05:17:05.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16 05:17:05 URL:https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm [357344123/357344123] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-16T05:17:05.170 INFO:teuthology.orchestra.run.smithi027.stderr:FINISHED --2024-04-16 05:17:05-- 2024-04-16T05:17:05.170 INFO:teuthology.orchestra.run.smithi027.stderr:Total wall clock time: 13s 2024-04-16T05:17:05.170 INFO:teuthology.orchestra.run.smithi027.stderr:Downloaded: 1 files, 341M in 12s (27.8 MB/s) 2024-04-16T05:17:05.173 INFO:teuthology.task.kernel:Installing kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on host.a... 2024-04-16T05:17:05.173 DEBUG:teuthology.orchestra.run.smithi027:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-16T05:17:06.422 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-16T05:17:06.422 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-16T05:17:07.323 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying... ######################################## 2024-04-16T05:17:07.379 INFO:teuthology.orchestra.run.smithi027.stdout:Preparing... ######################################## 2024-04-16T05:17:07.465 INFO:teuthology.orchestra.run.smithi027.stdout:Updating / installing... 2024-04-16T05:17:12.729 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-16T05:17:12.940 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-16T05:17:13.031 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T05:17:14.431 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T05:17:14.431 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-16T05:17:14.817 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-16T05:17:14.818 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:17:14.818 INFO:teuthology.orchestra.run.smithi122.stdout:Upgraded: 2024-04-16T05:17:14.818 INFO:teuthology.orchestra.run.smithi122.stdout: linux-firmware-20240219-143.el9.noarch 2024-04-16T05:17:14.818 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:17:14.818 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:17:15.090 INFO:teuthology.task.kernel:Downloading kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on host.b... 2024-04-16T05:17:15.090 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-16T05:17:15.321 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ 2024-04-16T05:17:15.321 DEBUG:teuthology.orchestra.run.smithi122:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-16T05:17:15.324 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi122.front.sepia.ceph.com to complete... 2024-04-16T05:17:15.839 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-6.9.0_rc3_g3f60ed14f7c1-1 ######################################## 2024-04-16T05:17:32.810 INFO:teuthology.orchestra.run.smithi027.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T05:17:33.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16 05:17:33 URL:https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm [357344123/357344123] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-16T05:17:33.831 INFO:teuthology.orchestra.run.smithi122.stderr:FINISHED --2024-04-16 05:17:33-- 2024-04-16T05:17:33.831 INFO:teuthology.orchestra.run.smithi122.stderr:Total wall clock time: 18s 2024-04-16T05:17:33.831 INFO:teuthology.orchestra.run.smithi122.stderr:Downloaded: 1 files, 341M in 18s (18.8 MB/s) 2024-04-16T05:17:33.833 INFO:teuthology.task.kernel:Installing kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on host.b... 2024-04-16T05:17:33.834 DEBUG:teuthology.orchestra.run.smithi122:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-16T05:17:35.875 INFO:teuthology.orchestra.run.smithi122.stdout:Verifying... ######################################## 2024-04-16T05:17:36.031 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing... ######################################## 2024-04-16T05:17:36.110 INFO:teuthology.orchestra.run.smithi122.stdout:Updating / installing... 2024-04-16T05:17:44.099 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-6.9.0_rc3_g3f60ed14f7c1-1 ######################################## 2024-04-16T05:17:56.668 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi027.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-16T05:17:56.668 DEBUG:teuthology.orchestra.run.smithi027:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-16T05:17:56.739 INFO:teuthology.orchestra.run.smithi027.stdout:/boot/System.map-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T05:17:56.740 INFO:teuthology.orchestra.run.smithi027.stdout:/boot/config-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T05:17:56.740 INFO:teuthology.orchestra.run.smithi027.stdout:/boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T05:17:56.740 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/System.map 2024-04-16T05:17:56.740 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/config 2024-04-16T05:17:56.740 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel 2024-04-16T05:17:56.740 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch 2024-04-16T05:17:56.740 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86 2024-04-16T05:17:56.740 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto 2024-04-16T05:17:56.740 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-16T05:17:56.740 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-16T05:17:56.740 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-16T05:17:56.740 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-16T05:17:56.741 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-16T05:17:56.741 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-16T05:17:56.741 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-16T05:17:56.741 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-16T05:17:56.741 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-16T05:17:56.741 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-16T05:17:56.741 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-16T05:17:56.741 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-16T05:17:56.741 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-16T05:17:56.741 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-16T05:17:56.741 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-16T05:17:56.741 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-16T05:17:56.741 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-16T05:17:56.741 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-16T05:17:56.742 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-16T05:17:56.742 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-16T05:17:56.742 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel 2024-04-16T05:17:56.742 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu 2024-04-16T05:17:56.744 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce 2024-04-16T05:17:56.744 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-16T05:17:56.744 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm 2024-04-16T05:17:56.744 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-16T05:17:56.744 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-16T05:17:56.744 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm.ko 2024-04-16T05:17:56.744 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto 2024-04-16T05:17:56.744 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ansi_cprng.ko 2024-04-16T05:17:56.744 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/anubis.ko 2024-04-16T05:17:56.744 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/arc4.ko 2024-04-16T05:17:56.744 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx 2024-04-16T05:17:56.744 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_memcpy.ko 2024-04-16T05:17:56.744 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_pq.ko 2024-04-16T05:17:56.745 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-16T05:17:56.745 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_tx.ko 2024-04-16T05:17:56.745 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_xor.ko 2024-04-16T05:17:56.745 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/raid6test.ko 2024-04-16T05:17:56.745 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authenc.ko 2024-04-16T05:17:56.745 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authencesn.ko 2024-04-16T05:17:56.745 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blake2b_generic.ko 2024-04-16T05:17:56.745 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_common.ko 2024-04-16T05:17:56.745 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_generic.ko 2024-04-16T05:17:56.745 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/camellia_generic.ko 2024-04-16T05:17:56.745 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast5_generic.ko 2024-04-16T05:17:56.745 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast6_generic.ko 2024-04-16T05:17:56.745 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast_common.ko 2024-04-16T05:17:56.745 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ccm.ko 2024-04-16T05:17:56.746 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cmac.ko 2024-04-16T05:17:56.746 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crc32_generic.ko 2024-04-16T05:17:56.746 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_engine.ko 2024-04-16T05:17:56.746 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_user.ko 2024-04-16T05:17:56.746 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cts.ko 2024-04-16T05:17:56.746 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/des_generic.ko 2024-04-16T05:17:56.746 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/dh_generic.ko 2024-04-16T05:17:56.746 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/echainiv.ko 2024-04-16T05:17:56.746 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/essiv.ko 2024-04-16T05:17:56.746 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/fcrypt.ko 2024-04-16T05:17:56.746 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/khazad.ko 2024-04-16T05:17:56.746 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4.ko 2024-04-16T05:17:56.746 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4hc.ko 2024-04-16T05:17:56.747 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/md4.ko 2024-04-16T05:17:56.747 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/michael_mic.ko 2024-04-16T05:17:56.747 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcbc.ko 2024-04-16T05:17:56.747 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcrypt.ko 2024-04-16T05:17:56.747 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/rmd160.ko 2024-04-16T05:17:56.747 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/seed.ko 2024-04-16T05:17:56.747 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/serpent_generic.ko 2024-04-16T05:17:56.747 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tcrypt.ko 2024-04-16T05:17:56.747 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tea.ko 2024-04-16T05:17:56.747 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_common.ko 2024-04-16T05:17:56.747 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_generic.ko 2024-04-16T05:17:56.747 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/vmac.ko 2024-04-16T05:17:56.747 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/wp512.ko 2024-04-16T05:17:56.747 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xcbc.ko 2024-04-16T05:17:56.748 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xor.ko 2024-04-16T05:17:56.748 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xxhash_generic.ko 2024-04-16T05:17:56.748 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers 2024-04-16T05:17:56.748 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi 2024-04-16T05:17:56.748 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-16T05:17:56.748 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_pad.ko 2024-04-16T05:17:56.748 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/ec_sys.ko 2024-04-16T05:17:56.748 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/platform_profile.ko 2024-04-16T05:17:56.748 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbs.ko 2024-04-16T05:17:56.748 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbshc.ko 2024-04-16T05:17:56.748 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/video.ko 2024-04-16T05:17:56.748 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata 2024-04-16T05:17:56.748 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/acard-ahci.ko 2024-04-16T05:17:56.748 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ahci_platform.ko 2024-04-16T05:17:56.749 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ata_generic.ko 2024-04-16T05:17:56.749 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/libahci_platform.ko 2024-04-16T05:17:56.749 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_acpi.ko 2024-04-16T05:17:56.749 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ali.ko 2024-04-16T05:17:56.749 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_amd.ko 2024-04-16T05:17:56.749 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_artop.ko 2024-04-16T05:17:56.749 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atiixp.ko 2024-04-16T05:17:56.749 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atp867x.ko 2024-04-16T05:17:56.749 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd640.ko 2024-04-16T05:17:56.749 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd64x.ko 2024-04-16T05:17:56.749 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cypress.ko 2024-04-16T05:17:56.749 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_efar.ko 2024-04-16T05:17:56.749 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt366.ko 2024-04-16T05:17:56.749 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt37x.ko 2024-04-16T05:17:56.749 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-16T05:17:56.750 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-16T05:17:56.750 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it8213.ko 2024-04-16T05:17:56.750 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it821x.ko 2024-04-16T05:17:56.750 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_jmicron.ko 2024-04-16T05:17:56.750 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_marvell.ko 2024-04-16T05:17:56.750 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_mpiix.ko 2024-04-16T05:17:56.750 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_netcell.ko 2024-04-16T05:17:56.750 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ninja32.ko 2024-04-16T05:17:56.750 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87410.ko 2024-04-16T05:17:56.751 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87415.ko 2024-04-16T05:17:56.751 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_oldpiix.ko 2024-04-16T05:17:56.751 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_opti.ko 2024-04-16T05:17:56.751 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_optidma.ko 2024-04-16T05:17:56.751 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pcmcia.ko 2024-04-16T05:17:56.751 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-16T05:17:56.751 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-16T05:17:56.751 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_piccolo.ko 2024-04-16T05:17:56.751 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_rdc.ko 2024-04-16T05:17:56.751 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sch.ko 2024-04-16T05:17:56.751 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_serverworks.ko 2024-04-16T05:17:56.751 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sil680.ko 2024-04-16T05:17:56.751 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sis.ko 2024-04-16T05:17:56.752 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sl82c105.ko 2024-04-16T05:17:56.752 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_triflex.ko 2024-04-16T05:17:56.752 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_via.ko 2024-04-16T05:17:56.752 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pdc_adma.ko 2024-04-16T05:17:56.752 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_inic162x.ko 2024-04-16T05:17:56.752 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_mv.ko 2024-04-16T05:17:56.752 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_nv.ko 2024-04-16T05:17:56.752 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_promise.ko 2024-04-16T05:17:56.752 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_qstor.ko 2024-04-16T05:17:56.752 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil.ko 2024-04-16T05:17:56.752 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil24.ko 2024-04-16T05:17:56.752 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sis.ko 2024-04-16T05:17:56.752 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_svw.ko 2024-04-16T05:17:56.752 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sx4.ko 2024-04-16T05:17:56.753 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_uli.ko 2024-04-16T05:17:56.753 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_via.ko 2024-04-16T05:17:56.753 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_vsc.ko 2024-04-16T05:17:56.753 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay 2024-04-16T05:17:56.753 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay/ks0108.ko 2024-04-16T05:17:56.753 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base 2024-04-16T05:17:56.753 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap 2024-04-16T05:17:56.753 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-16T05:17:56.753 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-16T05:17:56.753 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma 2024-04-16T05:17:56.753 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma/bcma.ko 2024-04-16T05:17:56.753 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block 2024-04-16T05:17:56.753 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe 2024-04-16T05:17:56.753 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe/aoe.ko 2024-04-16T05:17:56.754 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/brd.ko 2024-04-16T05:17:56.754 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd 2024-04-16T05:17:56.754 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd/drbd.ko 2024-04-16T05:17:56.755 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/floppy.ko 2024-04-16T05:17:56.755 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/loop.ko 2024-04-16T05:17:56.755 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx 2024-04-16T05:17:56.755 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-16T05:17:56.755 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/nbd.ko 2024-04-16T05:17:56.755 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk 2024-04-16T05:17:56.755 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk/null_blk.ko 2024-04-16T05:17:56.755 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/pktcdvd.ko 2024-04-16T05:17:56.755 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/rbd.ko 2024-04-16T05:17:56.755 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/virtio_blk.ko 2024-04-16T05:17:56.755 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback 2024-04-16T05:17:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-16T05:17:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkfront.ko 2024-04-16T05:17:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram 2024-04-16T05:17:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram/zram.ko 2024-04-16T05:17:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char 2024-04-16T05:17:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hangcheck-timer.ko 2024-04-16T05:17:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random 2024-04-16T05:17:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-16T05:17:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-16T05:17:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-16T05:17:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/via-rng.ko 2024-04-16T05:17:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-16T05:17:56.756 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi 2024-04-16T05:17:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-16T05:17:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-16T05:17:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-16T05:17:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-16T05:17:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-16T05:17:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-16T05:17:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/lp.ko 2024-04-16T05:17:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave 2024-04-16T05:17:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave/mwave.ko 2024-04-16T05:17:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ppdev.ko 2024-04-16T05:17:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tlclk.ko 2024-04-16T05:17:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm 2024-04-16T05:17:56.757 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm.ko 2024-04-16T05:17:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-16T05:17:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-16T05:17:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-16T05:17:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-16T05:17:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-16T05:17:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/uv_mmtimer.ko 2024-04-16T05:17:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/virtio_console.ko 2024-04-16T05:17:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq 2024-04-16T05:17:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-16T05:17:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-16T05:17:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-16T05:17:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-16T05:17:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-16T05:17:56.758 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-16T05:17:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto 2024-04-16T05:17:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp 2024-04-16T05:17:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-16T05:17:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp.ko 2024-04-16T05:17:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel 2024-04-16T05:17:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat 2024-04-16T05:17:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common 2024-04-16T05:17:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-16T05:17:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-16T05:17:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-16T05:17:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-aes.ko 2024-04-16T05:17:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-sha.ko 2024-04-16T05:17:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio 2024-04-16T05:17:56.759 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-16T05:17:56.760 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca 2024-04-16T05:17:56.760 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca/dca.ko 2024-04-16T05:17:56.760 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq 2024-04-16T05:17:56.760 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-16T05:17:56.760 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma 2024-04-16T05:17:56.760 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw 2024-04-16T05:17:56.760 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-16T05:17:56.760 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat 2024-04-16T05:17:56.760 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-16T05:17:56.760 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac 2024-04-16T05:17:56.760 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/amd64_edac.ko 2024-04-16T05:17:56.760 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/e752x_edac.ko 2024-04-16T05:17:56.760 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/edac_mce_amd.ko 2024-04-16T05:17:56.761 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3000_edac.ko 2024-04-16T05:17:56.761 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3200_edac.ko 2024-04-16T05:17:56.761 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5100_edac.ko 2024-04-16T05:17:56.761 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5400_edac.ko 2024-04-16T05:17:56.761 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7300_edac.ko 2024-04-16T05:17:56.761 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7core_edac.ko 2024-04-16T05:17:56.761 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i82975x_edac.ko 2024-04-16T05:17:56.761 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/ie31200_edac.ko 2024-04-16T05:17:56.761 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/sb_edac.ko 2024-04-16T05:17:56.761 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/x38_edac.ko 2024-04-16T05:17:56.761 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware 2024-04-16T05:17:56.761 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/edd.ko 2024-04-16T05:17:56.761 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-16T05:17:56.761 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio 2024-04-16T05:17:56.761 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-ich.ko 2024-04-16T05:17:56.762 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-16T05:17:56.762 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid 2024-04-16T05:17:56.762 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-appleir.ko 2024-04-16T05:17:56.762 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-aureal.ko 2024-04-16T05:17:56.762 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-axff.ko 2024-04-16T05:17:56.762 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-dr.ko 2024-04-16T05:17:56.762 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elecom.ko 2024-04-16T05:17:56.762 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elo.ko 2024-04-16T05:17:56.762 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-emsff.ko 2024-04-16T05:17:56.762 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gaff.ko 2024-04-16T05:17:56.762 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gt683r.ko 2024-04-16T05:17:56.762 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gyration.ko 2024-04-16T05:17:56.762 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-16T05:17:56.763 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-16T05:17:56.763 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtekff.ko 2024-04-16T05:17:56.763 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-hyperv.ko 2024-04-16T05:17:56.763 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-icade.ko 2024-04-16T05:17:56.763 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-keytouch.ko 2024-04-16T05:17:56.763 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-kye.ko 2024-04-16T05:17:56.763 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lcpower.ko 2024-04-16T05:17:56.763 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-led.ko 2024-04-16T05:17:56.763 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lenovo.ko 2024-04-16T05:17:56.763 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-16T05:17:56.763 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-16T05:17:56.763 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-multitouch.ko 2024-04-16T05:17:56.763 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ortek.ko 2024-04-16T05:17:56.763 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-penmount.ko 2024-04-16T05:17:56.763 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-petalynx.ko 2024-04-16T05:17:56.764 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-picolcd.ko 2024-04-16T05:17:56.764 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-pl.ko 2024-04-16T05:17:56.764 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-plantronics.ko 2024-04-16T05:17:56.764 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-primax.ko 2024-04-16T05:17:56.764 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-rmi.ko 2024-04-16T05:17:56.764 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-16T05:17:56.764 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-common.ko 2024-04-16T05:17:56.765 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-16T05:17:56.765 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-16T05:17:56.765 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-16T05:17:56.765 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-16T05:17:56.765 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-16T05:17:56.765 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-16T05:17:56.765 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-16T05:17:56.765 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-16T05:17:56.765 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-16T05:17:56.765 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat.ko 2024-04-16T05:17:56.765 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-saitek.ko 2024-04-16T05:17:56.765 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-samsung.ko 2024-04-16T05:17:56.765 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-16T05:17:56.766 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sjoy.ko 2024-04-16T05:17:56.766 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sony.ko 2024-04-16T05:17:56.766 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-speedlink.ko 2024-04-16T05:17:56.766 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-steelseries.ko 2024-04-16T05:17:56.766 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sunplus.ko 2024-04-16T05:17:56.766 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-16T05:17:56.766 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tivo.ko 2024-04-16T05:17:56.766 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tmff.ko 2024-04-16T05:17:56.766 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-topseed.ko 2024-04-16T05:17:56.766 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-twinhan.ko 2024-04-16T05:17:56.766 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-uclogic.ko 2024-04-16T05:17:56.766 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-waltop.ko 2024-04-16T05:17:56.766 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-wiimote.ko 2024-04-16T05:17:56.767 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-xinmo.ko 2024-04-16T05:17:56.767 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zpff.ko 2024-04-16T05:17:56.767 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zydacron.ko 2024-04-16T05:17:56.767 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/uhid.ko 2024-04-16T05:17:56.767 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/wacom.ko 2024-04-16T05:17:56.767 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv 2024-04-16T05:17:56.767 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_balloon.ko 2024-04-16T05:17:56.767 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_utils.ko 2024-04-16T05:17:56.767 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_vmbus.ko 2024-04-16T05:17:56.767 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon 2024-04-16T05:17:56.767 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru.ko 2024-04-16T05:17:56.767 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru3.ko 2024-04-16T05:17:56.767 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-16T05:17:56.767 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7414.ko 2024-04-16T05:17:56.768 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7418.ko 2024-04-16T05:17:56.768 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adc128d818.ko 2024-04-16T05:17:56.768 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1025.ko 2024-04-16T05:17:56.768 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1026.ko 2024-04-16T05:17:56.768 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1029.ko 2024-04-16T05:17:56.768 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1031.ko 2024-04-16T05:17:56.768 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm9240.ko 2024-04-16T05:17:56.768 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ads7828.ko 2024-04-16T05:17:56.768 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7410.ko 2024-04-16T05:17:56.769 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7411.ko 2024-04-16T05:17:56.769 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7462.ko 2024-04-16T05:17:56.769 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7470.ko 2024-04-16T05:17:56.769 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7475.ko 2024-04-16T05:17:56.769 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7x10.ko 2024-04-16T05:17:56.769 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/amc6821.ko 2024-04-16T05:17:56.769 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/applesmc.ko 2024-04-16T05:17:56.769 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asb100.ko 2024-04-16T05:17:56.769 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asc7621.ko 2024-04-16T05:17:56.769 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-16T05:17:56.770 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/atxp1.ko 2024-04-16T05:17:56.770 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/coretemp.ko 2024-04-16T05:17:56.770 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-16T05:17:56.770 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dme1737.ko 2024-04-16T05:17:56.770 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds1621.ko 2024-04-16T05:17:56.770 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds620.ko 2024-04-16T05:17:56.770 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc1403.ko 2024-04-16T05:17:56.770 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc6w201.ko 2024-04-16T05:17:56.770 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71805f.ko 2024-04-16T05:17:56.770 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71882fg.ko 2024-04-16T05:17:56.770 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f75375s.ko 2024-04-16T05:17:56.770 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fam15h_power.ko 2024-04-16T05:17:56.770 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fschmd.ko 2024-04-16T05:17:56.771 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g760a.ko 2024-04-16T05:17:56.771 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g762.ko 2024-04-16T05:17:56.771 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl518sm.ko 2024-04-16T05:17:56.771 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl520sm.ko 2024-04-16T05:17:56.771 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-16T05:17:56.771 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5500_temp.ko 2024-04-16T05:17:56.771 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5k_amb.ko 2024-04-16T05:17:56.771 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmaem.ko 2024-04-16T05:17:56.771 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmpex.ko 2024-04-16T05:17:56.771 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina209.ko 2024-04-16T05:17:56.771 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina2xx.ko 2024-04-16T05:17:56.771 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/it87.ko 2024-04-16T05:17:56.771 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k10temp.ko 2024-04-16T05:17:56.771 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k8temp.ko 2024-04-16T05:17:56.772 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lineage-pem.ko 2024-04-16T05:17:56.772 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm63.ko 2024-04-16T05:17:56.772 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm73.ko 2024-04-16T05:17:56.772 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm75.ko 2024-04-16T05:17:56.772 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm77.ko 2024-04-16T05:17:56.772 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm78.ko 2024-04-16T05:17:56.772 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm80.ko 2024-04-16T05:17:56.772 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm83.ko 2024-04-16T05:17:56.772 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm85.ko 2024-04-16T05:17:56.773 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm87.ko 2024-04-16T05:17:56.773 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm90.ko 2024-04-16T05:17:56.773 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm92.ko 2024-04-16T05:17:56.773 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm93.ko 2024-04-16T05:17:56.773 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95234.ko 2024-04-16T05:17:56.773 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95241.ko 2024-04-16T05:17:56.773 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95245.ko 2024-04-16T05:17:56.773 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc2945.ko 2024-04-16T05:17:56.773 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4151.ko 2024-04-16T05:17:56.773 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4215.ko 2024-04-16T05:17:56.774 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4222.ko 2024-04-16T05:17:56.774 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4245.ko 2024-04-16T05:17:56.774 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4260.ko 2024-04-16T05:17:56.774 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4261.ko 2024-04-16T05:17:56.774 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max16065.ko 2024-04-16T05:17:56.774 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1619.ko 2024-04-16T05:17:56.774 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1668.ko 2024-04-16T05:17:56.774 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max197.ko 2024-04-16T05:17:56.774 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6639.ko 2024-04-16T05:17:56.774 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6650.ko 2024-04-16T05:17:56.775 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6697.ko 2024-04-16T05:17:56.775 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/mcp3021.ko 2024-04-16T05:17:56.775 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6683.ko 2024-04-16T05:17:56.775 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775-core.ko 2024-04-16T05:17:56.776 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775.ko 2024-04-16T05:17:56.776 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct7802.ko 2024-04-16T05:17:56.776 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87360.ko 2024-04-16T05:17:56.776 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87427.ko 2024-04-16T05:17:56.776 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pcf8591.ko 2024-04-16T05:17:56.776 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus 2024-04-16T05:17:56.776 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-16T05:17:56.776 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-16T05:17:56.776 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-16T05:17:56.776 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-16T05:17:56.776 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-16T05:17:56.776 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-16T05:17:56.776 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-16T05:17:56.776 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-16T05:17:56.777 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-16T05:17:56.777 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-16T05:17:56.777 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-16T05:17:56.777 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-16T05:17:56.777 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/powr1220.ko 2024-04-16T05:17:56.777 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5627.ko 2024-04-16T05:17:56.777 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5636.ko 2024-04-16T05:17:56.777 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-16T05:17:56.778 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht15.ko 2024-04-16T05:17:56.778 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht21.ko 2024-04-16T05:17:56.778 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/shtc1.ko 2024-04-16T05:17:56.778 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sis5595.ko 2024-04-16T05:17:56.778 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47b397.ko 2024-04-16T05:17:56.779 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m1.ko 2024-04-16T05:17:56.779 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m192.ko 2024-04-16T05:17:56.779 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/thmc50.ko 2024-04-16T05:17:56.779 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp102.ko 2024-04-16T05:17:56.779 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp103.ko 2024-04-16T05:17:56.779 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp401.ko 2024-04-16T05:17:56.779 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp421.ko 2024-04-16T05:17:56.779 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via-cputemp.ko 2024-04-16T05:17:56.779 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via686a.ko 2024-04-16T05:17:56.779 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt1211.ko 2024-04-16T05:17:56.779 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt8231.ko 2024-04-16T05:17:56.780 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627ehf.ko 2024-04-16T05:17:56.780 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627hf.ko 2024-04-16T05:17:56.780 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83781d.ko 2024-04-16T05:17:56.780 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83791d.ko 2024-04-16T05:17:56.780 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83792d.ko 2024-04-16T05:17:56.780 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83793.ko 2024-04-16T05:17:56.780 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83795.ko 2024-04-16T05:17:56.780 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l785ts.ko 2024-04-16T05:17:56.780 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l786ng.ko 2024-04-16T05:17:56.780 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c 2024-04-16T05:17:56.780 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos 2024-04-16T05:17:56.780 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-16T05:17:56.780 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-16T05:17:56.781 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses 2024-04-16T05:17:56.781 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-16T05:17:56.781 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-16T05:17:56.781 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-16T05:17:56.781 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-16T05:17:56.781 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-16T05:17:56.781 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-16T05:17:56.781 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-16T05:17:56.781 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-16T05:17:56.781 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-16T05:17:56.781 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-16T05:17:56.781 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-16T05:17:56.781 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-16T05:17:56.781 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-16T05:17:56.782 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-16T05:17:56.782 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-16T05:17:56.782 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-16T05:17:56.782 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-16T05:17:56.782 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-16T05:17:56.782 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-16T05:17:56.782 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-16T05:17:56.782 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-16T05:17:56.782 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-16T05:17:56.782 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-16T05:17:56.782 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-dev.ko 2024-04-16T05:17:56.782 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-mux.ko 2024-04-16T05:17:56.782 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-16T05:17:56.783 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-smbus.ko 2024-04-16T05:17:56.783 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-stub.ko 2024-04-16T05:17:56.783 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband 2024-04-16T05:17:56.783 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core 2024-04-16T05:17:56.783 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-16T05:17:56.783 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_core.ko 2024-04-16T05:17:56.783 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-16T05:17:56.783 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-16T05:17:56.783 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-16T05:17:56.783 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-16T05:17:56.783 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-16T05:17:56.783 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw 2024-04-16T05:17:56.783 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4 2024-04-16T05:17:56.783 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-16T05:17:56.784 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4 2024-04-16T05:17:56.784 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-16T05:17:56.784 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5 2024-04-16T05:17:56.784 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-16T05:17:56.784 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca 2024-04-16T05:17:56.784 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-16T05:17:56.784 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp 2024-04-16T05:17:56.784 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib 2024-04-16T05:17:56.784 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-16T05:17:56.784 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser 2024-04-16T05:17:56.784 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-16T05:17:56.784 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert 2024-04-16T05:17:56.784 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-16T05:17:56.784 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp 2024-04-16T05:17:56.785 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-16T05:17:56.785 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt 2024-04-16T05:17:56.785 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-16T05:17:56.785 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input 2024-04-16T05:17:56.785 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport 2024-04-16T05:17:56.785 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-16T05:17:56.785 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-16T05:17:56.786 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/gameport.ko 2024-04-16T05:17:56.786 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/joydev.ko 2024-04-16T05:17:56.786 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard 2024-04-16T05:17:56.786 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-16T05:17:56.786 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse 2024-04-16T05:17:56.786 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/appletouch.ko 2024-04-16T05:17:56.786 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/bcm5974.ko 2024-04-16T05:17:56.786 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/cyapatp.ko 2024-04-16T05:17:56.786 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-16T05:17:56.786 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/sermouse.ko 2024-04-16T05:17:56.786 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-16T05:17:56.786 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-16T05:17:56.786 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-16T05:17:56.787 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4 2024-04-16T05:17:56.787 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-16T05:17:56.787 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio 2024-04-16T05:17:56.787 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/altera_ps2.ko 2024-04-16T05:17:56.787 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/arc_ps2.ko 2024-04-16T05:17:56.787 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-16T05:17:56.787 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/serio_raw.ko 2024-04-16T05:17:56.787 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/sparse-keymap.ko 2024-04-16T05:17:56.787 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds 2024-04-16T05:17:56.787 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-blinkm.ko 2024-04-16T05:17:56.787 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3530.ko 2024-04-16T05:17:56.787 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lp3944.ko 2024-04-16T05:17:56.787 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-ss4200.ko 2024-04-16T05:17:56.787 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger 2024-04-16T05:17:56.788 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-16T05:17:56.788 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-16T05:17:56.788 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-16T05:17:56.788 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-16T05:17:56.788 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-16T05:17:56.788 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-16T05:17:56.788 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-16T05:17:56.788 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-16T05:17:56.788 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-16T05:17:56.789 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md 2024-04-16T05:17:56.789 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache 2024-04-16T05:17:56.789 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache/bcache.ko 2024-04-16T05:17:56.789 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-bio-prison.ko 2024-04-16T05:17:56.789 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache-smq.ko 2024-04-16T05:17:56.789 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache.ko 2024-04-16T05:17:56.789 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-crypt.ko 2024-04-16T05:17:56.789 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-delay.ko 2024-04-16T05:17:56.789 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-flakey.ko 2024-04-16T05:17:56.789 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-log-userspace.ko 2024-04-16T05:17:56.789 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-multipath.ko 2024-04-16T05:17:56.789 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-queue-length.ko 2024-04-16T05:17:56.789 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-raid.ko 2024-04-16T05:17:56.789 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-round-robin.ko 2024-04-16T05:17:56.790 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-service-time.ko 2024-04-16T05:17:56.790 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-switch.ko 2024-04-16T05:17:56.790 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-thin-pool.ko 2024-04-16T05:17:56.790 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-verity.ko 2024-04-16T05:17:56.790 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data 2024-04-16T05:17:56.790 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-16T05:17:56.790 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid0.ko 2024-04-16T05:17:56.790 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid1.ko 2024-04-16T05:17:56.790 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid10.ko 2024-04-16T05:17:56.790 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid456.ko 2024-04-16T05:17:56.790 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message 2024-04-16T05:17:56.790 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion 2024-04-16T05:17:56.790 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptbase.ko 2024-04-16T05:17:56.791 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptctl.ko 2024-04-16T05:17:56.791 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptfc.ko 2024-04-16T05:17:56.791 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptlan.ko 2024-04-16T05:17:56.791 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptsas.ko 2024-04-16T05:17:56.791 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptscsih.ko 2024-04-16T05:17:56.791 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptspi.ko 2024-04-16T05:17:56.791 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd 2024-04-16T05:17:56.791 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_ich.ko 2024-04-16T05:17:56.791 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_sch.ko 2024-04-16T05:17:56.791 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/mfd-core.ko 2024-04-16T05:17:56.791 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/sm501.ko 2024-04-16T05:17:56.791 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/viperboard.ko 2024-04-16T05:17:56.791 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/vx855.ko 2024-04-16T05:17:56.791 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/wl1273-core.ko 2024-04-16T05:17:56.792 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc 2024-04-16T05:17:56.792 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl 2024-04-16T05:17:56.792 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-16T05:17:56.792 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds9802als.ko 2024-04-16T05:17:56.792 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds990x.ko 2024-04-16T05:17:56.792 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/bh1770glc.ko 2024-04-16T05:17:56.792 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710 2024-04-16T05:17:56.792 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710/cb710.ko 2024-04-16T05:17:56.792 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom 2024-04-16T05:17:56.792 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/at24.ko 2024-04-16T05:17:56.792 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-16T05:17:56.792 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/max6875.ko 2024-04-16T05:17:56.792 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/enclosure.ko 2024-04-16T05:17:56.792 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/hpilo.ko 2024-04-16T05:17:56.793 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29003.ko 2024-04-16T05:17:56.793 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29020.ko 2024-04-16T05:17:56.793 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d 2024-04-16T05:17:56.793 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-16T05:17:56.793 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-16T05:17:56.793 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei 2024-04-16T05:17:56.793 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-me.ko 2024-04-16T05:17:56.793 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-txe.ko 2024-04-16T05:17:56.793 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei.ko 2024-04-16T05:17:56.794 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru 2024-04-16T05:17:56.794 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-16T05:17:56.794 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp 2024-04-16T05:17:56.794 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-16T05:17:56.794 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-16T05:17:56.794 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-16T05:17:56.794 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_7xx1.ko 2024-04-16T05:17:56.794 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_core.ko 2024-04-16T05:17:56.794 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tsl2550.ko 2024-04-16T05:17:56.794 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_balloon.ko 2024-04-16T05:17:56.795 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci 2024-04-16T05:17:56.795 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-16T05:17:56.795 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net 2024-04-16T05:17:56.795 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding 2024-04-16T05:17:56.795 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding/bonding.ko 2024-04-16T05:17:56.795 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa 2024-04-16T05:17:56.795 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53 2024-04-16T05:17:56.795 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-16T05:17:56.795 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-16T05:17:56.795 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-16T05:17:56.795 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx 2024-04-16T05:17:56.795 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-16T05:17:56.795 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dummy.ko 2024-04-16T05:17:56.796 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/eql.ko 2024-04-16T05:17:56.796 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet 2024-04-16T05:17:56.796 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com 2024-04-16T05:17:56.796 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-16T05:17:56.796 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-16T05:17:56.796 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-16T05:17:56.796 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-16T05:17:56.796 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390 2024-04-16T05:17:56.797 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-16T05:17:56.797 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-16T05:17:56.797 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-16T05:17:56.797 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-16T05:17:56.797 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec 2024-04-16T05:17:56.797 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-16T05:17:56.797 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere 2024-04-16T05:17:56.797 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-16T05:17:56.797 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon 2024-04-16T05:17:56.797 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-16T05:17:56.797 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera 2024-04-16T05:17:56.797 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-16T05:17:56.797 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd 2024-04-16T05:17:56.797 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-16T05:17:56.798 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-16T05:17:56.798 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-16T05:17:56.798 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros 2024-04-16T05:17:56.798 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx 2024-04-16T05:17:56.798 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-16T05:17:56.798 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-16T05:17:56.798 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-16T05:17:56.798 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-16T05:17:56.798 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-16T05:17:56.798 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx 2024-04-16T05:17:56.798 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-16T05:17:56.798 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-16T05:17:56.798 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom 2024-04-16T05:17:56.799 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-16T05:17:56.799 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-16T05:17:56.799 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-16T05:17:56.799 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-16T05:17:56.799 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-16T05:17:56.799 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet 2024-04-16T05:17:56.799 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-16T05:17:56.799 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-16T05:17:56.799 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade 2024-04-16T05:17:56.799 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna 2024-04-16T05:17:56.799 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-16T05:17:56.799 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio 2024-04-16T05:17:56.799 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-16T05:17:56.799 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-16T05:17:56.800 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-16T05:17:56.800 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-16T05:17:56.800 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-16T05:17:56.800 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-16T05:17:56.800 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-16T05:17:56.800 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-16T05:17:56.800 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-16T05:17:56.800 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-16T05:17:56.800 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco 2024-04-16T05:17:56.800 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic 2024-04-16T05:17:56.800 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-16T05:17:56.800 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec 2024-04-16T05:17:56.801 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip 2024-04-16T05:17:56.801 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-16T05:17:56.801 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-16T05:17:56.801 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-16T05:17:56.801 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-16T05:17:56.801 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-16T05:17:56.801 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-16T05:17:56.801 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink 2024-04-16T05:17:56.801 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-16T05:17:56.801 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-16T05:17:56.801 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dnet.ko 2024-04-16T05:17:56.801 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex 2024-04-16T05:17:56.801 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet 2024-04-16T05:17:56.801 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-16T05:17:56.802 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ethoc.ko 2024-04-16T05:17:56.802 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fealnx.ko 2024-04-16T05:17:56.802 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel 2024-04-16T05:17:56.802 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-16T05:17:56.802 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000 2024-04-16T05:17:56.802 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-16T05:17:56.802 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e 2024-04-16T05:17:56.802 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-16T05:17:56.802 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k 2024-04-16T05:17:56.802 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-16T05:17:56.802 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e 2024-04-16T05:17:56.802 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-16T05:17:56.802 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf 2024-04-16T05:17:56.802 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-16T05:17:56.803 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb 2024-04-16T05:17:56.803 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-16T05:17:56.803 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf 2024-04-16T05:17:56.803 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-16T05:17:56.803 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-16T05:17:56.803 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-16T05:17:56.803 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-16T05:17:56.803 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-16T05:17:56.804 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/jme.ko 2024-04-16T05:17:56.804 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell 2024-04-16T05:17:56.804 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-16T05:17:56.804 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-16T05:17:56.804 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-16T05:17:56.804 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox 2024-04-16T05:17:56.804 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-16T05:17:56.804 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-16T05:17:56.804 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-16T05:17:56.804 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-16T05:17:56.804 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-16T05:17:56.804 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-16T05:17:56.804 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel 2024-04-16T05:17:56.804 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-16T05:17:56.805 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom 2024-04-16T05:17:56.805 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-16T05:17:56.805 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-16T05:17:56.805 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi 2024-04-16T05:17:56.805 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-16T05:17:56.805 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-16T05:17:56.805 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion 2024-04-16T05:17:56.805 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-16T05:17:56.805 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia 2024-04-16T05:17:56.805 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-16T05:17:56.805 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines 2024-04-16T05:17:56.805 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-16T05:17:56.805 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-16T05:17:56.805 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic 2024-04-16T05:17:56.806 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-16T05:17:56.806 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-16T05:17:56.806 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-16T05:17:56.806 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-16T05:17:56.806 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-16T05:17:56.806 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc 2024-04-16T05:17:56.807 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-16T05:17:56.807 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek 2024-04-16T05:17:56.807 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-16T05:17:56.807 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-16T05:17:56.807 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-16T05:17:56.807 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-16T05:17:56.807 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker 2024-04-16T05:17:56.807 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-16T05:17:56.807 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc 2024-04-16T05:17:56.807 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-16T05:17:56.807 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan 2024-04-16T05:17:56.807 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-16T05:17:56.807 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis 2024-04-16T05:17:56.807 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-16T05:17:56.808 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-16T05:17:56.808 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc 2024-04-16T05:17:56.808 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-16T05:17:56.808 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-16T05:17:56.808 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-16T05:17:56.808 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-16T05:17:56.808 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro 2024-04-16T05:17:56.808 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-16T05:17:56.808 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-16T05:17:56.808 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-16T05:17:56.808 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun 2024-04-16T05:17:56.808 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-16T05:17:56.808 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-16T05:17:56.809 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-16T05:17:56.809 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-16T05:17:56.809 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti 2024-04-16T05:17:56.809 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-16T05:17:56.809 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti 2024-04-16T05:17:56.809 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-16T05:17:56.809 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via 2024-04-16T05:17:56.809 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-16T05:17:56.809 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-16T05:17:56.809 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet 2024-04-16T05:17:56.809 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-16T05:17:56.809 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-16T05:17:56.809 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom 2024-04-16T05:17:56.809 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-16T05:17:56.809 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/geneve.ko 2024-04-16T05:17:56.810 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv 2024-04-16T05:17:56.810 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-16T05:17:56.810 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154 2024-04-16T05:17:56.810 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-16T05:17:56.810 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ifb.ko 2024-04-16T05:17:56.810 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan 2024-04-16T05:17:56.810 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-16T05:17:56.810 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvlan.ko 2024-04-16T05:17:56.810 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvtap.ko 2024-04-16T05:17:56.810 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio 2024-04-16T05:17:56.810 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio.ko 2024-04-16T05:17:56.810 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-16T05:17:56.810 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-16T05:17:56.811 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-16T05:17:56.811 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mii.ko 2024-04-16T05:17:56.811 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/net_failover.ko 2024-04-16T05:17:56.811 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/netconsole.ko 2024-04-16T05:17:56.811 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/nlmon.ko 2024-04-16T05:17:56.811 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs 2024-04-16T05:17:56.811 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-16T05:17:56.811 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-16T05:17:56.811 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy 2024-04-16T05:17:56.811 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/amd.ko 2024-04-16T05:17:56.811 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-16T05:17:56.811 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-16T05:17:56.811 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-16T05:17:56.811 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm87xx.ko 2024-04-16T05:17:56.811 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/broadcom.ko 2024-04-16T05:17:56.812 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/cicada.ko 2024-04-16T05:17:56.812 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/davicom.ko 2024-04-16T05:17:56.812 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/dp83640.ko 2024-04-16T05:17:56.812 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/et1011c.ko 2024-04-16T05:17:56.812 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/icplus.ko 2024-04-16T05:17:56.812 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/lxt.ko 2024-04-16T05:17:56.812 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/marvell.ko 2024-04-16T05:17:56.812 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/micrel.ko 2024-04-16T05:17:56.812 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/national.ko 2024-04-16T05:17:56.813 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/phylink.ko 2024-04-16T05:17:56.813 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/qsemi.ko 2024-04-16T05:17:56.813 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/realtek.ko 2024-04-16T05:17:56.813 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/smsc.ko 2024-04-16T05:17:56.813 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/ste10Xp.ko 2024-04-16T05:17:56.813 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/vitesse.ko 2024-04-16T05:17:56.813 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp 2024-04-16T05:17:56.813 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-16T05:17:56.813 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_async.ko 2024-04-16T05:17:56.813 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-16T05:17:56.813 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-16T05:17:56.813 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-16T05:17:56.813 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-16T05:17:56.813 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppoe.ko 2024-04-16T05:17:56.814 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppox.ko 2024-04-16T05:17:56.814 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pptp.ko 2024-04-16T05:17:56.814 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip 2024-04-16T05:17:56.814 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slhc.ko 2024-04-16T05:17:56.814 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slip.ko 2024-04-16T05:17:56.814 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/sungem_phy.ko 2024-04-16T05:17:56.814 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tap.ko 2024-04-16T05:17:56.814 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team 2024-04-16T05:17:56.814 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team.ko 2024-04-16T05:17:56.814 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-16T05:17:56.814 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-16T05:17:56.814 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-16T05:17:56.814 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_random.ko 2024-04-16T05:17:56.815 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-16T05:17:56.815 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tun.ko 2024-04-16T05:17:56.815 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/veth.ko 2024-04-16T05:17:56.815 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/virtio_net.ko 2024-04-16T05:17:56.815 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3 2024-04-16T05:17:56.815 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-16T05:17:56.815 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan 2024-04-16T05:17:56.815 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan/vxlan.ko 2024-04-16T05:17:56.815 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback 2024-04-16T05:17:56.815 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-16T05:17:56.815 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netfront.ko 2024-04-16T05:17:56.815 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb 2024-04-16T05:17:56.815 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb/ntb.ko 2024-04-16T05:17:56.815 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme 2024-04-16T05:17:56.815 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host 2024-04-16T05:17:56.816 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme-core.ko 2024-04-16T05:17:56.816 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme.ko 2024-04-16T05:17:56.816 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport 2024-04-16T05:17:56.816 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport.ko 2024-04-16T05:17:56.816 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_cs.ko 2024-04-16T05:17:56.816 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_pc.ko 2024-04-16T05:17:56.816 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_serial.ko 2024-04-16T05:17:56.816 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci 2024-04-16T05:17:56.816 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug 2024-04-16T05:17:56.817 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-16T05:17:56.817 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie 2024-04-16T05:17:56.817 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-16T05:17:56.818 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/xen-pcifront.ko 2024-04-16T05:17:56.818 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia 2024-04-16T05:17:56.818 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/i82092.ko 2024-04-16T05:17:56.818 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pd6729.ko 2024-04-16T05:17:56.818 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-16T05:17:56.818 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl 2024-04-16T05:17:56.818 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel 2024-04-16T05:17:56.818 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-16T05:17:56.818 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform 2024-04-16T05:17:56.818 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86 2024-04-16T05:17:56.818 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-16T05:17:56.818 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acerhdf.ko 2024-04-16T05:17:56.818 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-16T05:17:56.818 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-16T05:17:56.819 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-16T05:17:56.819 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-16T05:17:56.819 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-16T05:17:56.819 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-16T05:17:56.819 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-16T05:17:56.819 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-16T05:17:56.819 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-16T05:17:56.819 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/hdaps.ko 2024-04-16T05:17:56.819 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel 2024-04-16T05:17:56.819 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-16T05:17:56.819 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel_ips.ko 2024-04-16T05:17:56.819 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-16T05:17:56.819 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-16T05:17:56.820 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-16T05:17:56.820 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-16T05:17:56.820 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-16T05:17:56.820 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-16T05:17:56.820 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-16T05:17:56.820 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-16T05:17:56.820 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-16T05:17:56.820 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi.ko 2024-04-16T05:17:56.820 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap 2024-04-16T05:17:56.820 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-16T05:17:56.820 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-16T05:17:56.820 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps 2024-04-16T05:17:56.820 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients 2024-04-16T05:17:56.820 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-16T05:17:56.821 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-16T05:17:56.821 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps_parport.ko 2024-04-16T05:17:56.821 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/pps_core.ko 2024-04-16T05:17:56.821 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp 2024-04-16T05:17:56.821 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp.ko 2024-04-16T05:17:56.821 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp_kvm.ko 2024-04-16T05:17:56.821 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras 2024-04-16T05:17:56.821 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd 2024-04-16T05:17:56.821 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl 2024-04-16T05:17:56.821 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-16T05:17:56.821 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/fmpm.ko 2024-04-16T05:17:56.821 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc 2024-04-16T05:17:56.821 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-16T05:17:56.821 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-16T05:17:56.822 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-16T05:17:56.822 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-16T05:17:56.822 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-16T05:17:56.822 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-16T05:17:56.822 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-16T05:17:56.822 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-16T05:17:56.822 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-16T05:17:56.822 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-16T05:17:56.822 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-em3027.ko 2024-04-16T05:17:56.822 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-16T05:17:56.822 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-16T05:17:56.822 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-16T05:17:56.822 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-16T05:17:56.823 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-16T05:17:56.823 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-16T05:17:56.823 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max6900.ko 2024-04-16T05:17:56.823 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-16T05:17:56.823 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-16T05:17:56.823 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-16T05:17:56.823 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-16T05:17:56.823 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-16T05:17:56.823 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-16T05:17:56.823 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-16T05:17:56.823 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-16T05:17:56.823 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-16T05:17:56.824 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-16T05:17:56.824 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-16T05:17:56.824 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-16T05:17:56.824 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-x1205.ko 2024-04-16T05:17:56.824 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi 2024-04-16T05:17:56.824 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-9xxx.ko 2024-04-16T05:17:56.824 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-sas.ko 2024-04-16T05:17:56.824 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-xxxx.ko 2024-04-16T05:17:56.824 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/BusLogic.ko 2024-04-16T05:17:56.824 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/a100u2w.ko 2024-04-16T05:17:56.824 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid 2024-04-16T05:17:56.824 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-16T05:17:56.825 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/advansys.ko 2024-04-16T05:17:56.825 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx 2024-04-16T05:17:56.825 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-16T05:17:56.825 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-16T05:17:56.825 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/am53c974.ko 2024-04-16T05:17:56.825 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr 2024-04-16T05:17:56.825 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-16T05:17:56.825 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/atp870u.ko 2024-04-16T05:17:56.825 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi 2024-04-16T05:17:56.826 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-16T05:17:56.826 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa 2024-04-16T05:17:56.826 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa/bfa.ko 2024-04-16T05:17:56.826 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc 2024-04-16T05:17:56.826 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-16T05:17:56.826 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i 2024-04-16T05:17:56.826 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-16T05:17:56.826 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ch.ko 2024-04-16T05:17:56.826 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor 2024-04-16T05:17:56.826 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-16T05:17:56.826 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi 2024-04-16T05:17:56.826 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-16T05:17:56.826 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-16T05:17:56.827 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-16T05:17:56.827 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-16T05:17:56.827 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-16T05:17:56.827 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dc395x.ko 2024-04-16T05:17:56.827 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler 2024-04-16T05:17:56.827 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-16T05:17:56.827 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-16T05:17:56.827 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-16T05:17:56.827 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-16T05:17:56.827 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dmx3191d.ko 2024-04-16T05:17:56.827 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r 2024-04-16T05:17:56.827 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-16T05:17:56.827 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esp_scsi.ko 2024-04-16T05:17:56.827 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe 2024-04-16T05:17:56.828 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-16T05:17:56.828 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-16T05:17:56.828 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic 2024-04-16T05:17:56.828 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic/fnic.ko 2024-04-16T05:17:56.828 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hpsa.ko 2024-04-16T05:17:56.828 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hptiop.ko 2024-04-16T05:17:56.828 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hv_storvsc.ko 2024-04-16T05:17:56.828 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/initio.ko 2024-04-16T05:17:56.828 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ipr.ko 2024-04-16T05:17:56.828 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ips.ko 2024-04-16T05:17:56.828 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci 2024-04-16T05:17:56.828 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci/isci.ko 2024-04-16T05:17:56.828 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-16T05:17:56.828 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-16T05:17:56.829 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc 2024-04-16T05:17:56.829 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc/libfc.ko 2024-04-16T05:17:56.829 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi.ko 2024-04-16T05:17:56.829 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-16T05:17:56.829 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas 2024-04-16T05:17:56.829 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas/libsas.ko 2024-04-16T05:17:56.829 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc 2024-04-16T05:17:56.829 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-16T05:17:56.829 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid 2024-04-16T05:17:56.829 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid.ko 2024-04-16T05:17:56.829 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-16T05:17:56.829 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-16T05:17:56.829 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-16T05:17:56.829 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas 2024-04-16T05:17:56.830 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-16T05:17:56.830 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas 2024-04-16T05:17:56.830 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-16T05:17:56.830 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvumi.ko 2024-04-16T05:17:56.830 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001 2024-04-16T05:17:56.830 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-16T05:17:56.830 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pmcraid.ko 2024-04-16T05:17:56.830 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla1280.ko 2024-04-16T05:17:56.830 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx 2024-04-16T05:17:56.830 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-16T05:17:56.830 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-16T05:17:56.830 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx 2024-04-16T05:17:56.830 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-16T05:17:56.830 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/raid_class.ko 2024-04-16T05:17:56.830 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_debug.ko 2024-04-16T05:17:56.831 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-16T05:17:56.831 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-16T05:17:56.831 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-16T05:17:56.831 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-16T05:17:56.831 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-16T05:17:56.831 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ses.ko 2024-04-16T05:17:56.831 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/st.ko 2024-04-16T05:17:56.831 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/stex.ko 2024-04-16T05:17:56.831 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2 2024-04-16T05:17:56.831 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-16T05:17:56.831 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/virtio_scsi.ko 2024-04-16T05:17:56.831 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-16T05:17:56.831 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/wd719x.ko 2024-04-16T05:17:56.832 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb 2024-04-16T05:17:56.832 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb/ssb.ko 2024-04-16T05:17:56.832 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target 2024-04-16T05:17:56.832 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi 2024-04-16T05:17:56.832 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-16T05:17:56.832 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback 2024-04-16T05:17:56.832 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-16T05:17:56.832 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_file.ko 2024-04-16T05:17:56.832 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_iblock.ko 2024-04-16T05:17:56.832 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_mod.ko 2024-04-16T05:17:56.832 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_pscsi.ko 2024-04-16T05:17:56.832 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_user.ko 2024-04-16T05:17:56.832 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc 2024-04-16T05:17:56.833 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-16T05:17:56.833 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal 2024-04-16T05:17:56.833 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel 2024-04-16T05:17:56.833 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal 2024-04-16T05:17:56.833 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-16T05:17:56.833 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-16T05:17:56.833 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-16T05:17:56.833 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-16T05:17:56.833 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-16T05:17:56.833 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-16T05:17:56.833 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-16T05:17:56.833 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-16T05:17:56.833 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-16T05:17:56.833 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-16T05:17:56.834 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-16T05:17:56.834 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-16T05:17:56.834 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-16T05:17:56.834 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-16T05:17:56.834 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-16T05:17:56.834 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-16T05:17:56.834 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-16T05:17:56.834 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-16T05:17:56.834 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty 2024-04-16T05:17:56.834 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless 2024-04-16T05:17:56.834 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-16T05:17:56.834 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_gsm.ko 2024-04-16T05:17:56.834 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_hdlc.ko 2024-04-16T05:17:56.835 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/nozomi.ko 2024-04-16T05:17:56.835 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial 2024-04-16T05:17:56.835 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250 2024-04-16T05:17:56.835 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-16T05:17:56.835 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/arc_uart.ko 2024-04-16T05:17:56.835 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm 2024-04-16T05:17:56.835 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-16T05:17:56.835 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs 2024-04-16T05:17:56.835 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core 2024-04-16T05:17:56.835 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-16T05:17:56.835 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host 2024-04-16T05:17:56.835 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-16T05:17:56.835 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio 2024-04-16T05:17:56.835 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio.ko 2024-04-16T05:17:56.836 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_aec.ko 2024-04-16T05:17:56.836 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_cif.ko 2024-04-16T05:17:56.836 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_pci_generic.ko 2024-04-16T05:17:56.836 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_sercos3.ko 2024-04-16T05:17:56.836 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb 2024-04-16T05:17:56.836 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class 2024-04-16T05:17:56.836 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-acm.ko 2024-04-16T05:17:56.836 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-16T05:17:56.836 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usblp.ko 2024-04-16T05:17:56.836 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usbtmc.ko 2024-04-16T05:17:56.836 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host 2024-04-16T05:17:56.836 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-16T05:17:56.836 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image 2024-04-16T05:17:56.836 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/mdc800.ko 2024-04-16T05:17:56.837 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/microtek.ko 2024-04-16T05:17:56.837 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc 2024-04-16T05:17:56.837 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/adutux.ko 2024-04-16T05:17:56.837 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/appledisplay.ko 2024-04-16T05:17:56.837 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi26.ko 2024-04-16T05:17:56.837 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi62.ko 2024-04-16T05:17:56.837 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ezusb.ko 2024-04-16T05:17:56.837 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/idmouse.ko 2024-04-16T05:17:56.837 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/iowarrior.ko 2024-04-16T05:17:56.837 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-16T05:17:56.837 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ldusb.ko 2024-04-16T05:17:56.837 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/legousbtower.ko 2024-04-16T05:17:56.837 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga 2024-04-16T05:17:56.838 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-16T05:17:56.838 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-16T05:17:56.838 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usb3503.ko 2024-04-16T05:17:56.838 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usblcd.ko 2024-04-16T05:17:56.838 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-16T05:17:56.838 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/uss720.ko 2024-04-16T05:17:56.838 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/yurex.ko 2024-04-16T05:17:56.838 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy 2024-04-16T05:17:56.838 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-generic.ko 2024-04-16T05:17:56.838 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial 2024-04-16T05:17:56.838 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/aircable.ko 2024-04-16T05:17:56.838 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ark3116.ko 2024-04-16T05:17:56.838 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-16T05:17:56.839 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ch341.ko 2024-04-16T05:17:56.839 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cp210x.ko 2024-04-16T05:17:56.839 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cyberjack.ko 2024-04-16T05:17:56.839 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-16T05:17:56.839 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-16T05:17:56.839 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/empeg.ko 2024-04-16T05:17:56.839 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-16T05:17:56.839 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-16T05:17:56.839 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-16T05:17:56.839 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_ti.ko 2024-04-16T05:17:56.839 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipaq.ko 2024-04-16T05:17:56.839 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipw.ko 2024-04-16T05:17:56.839 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ir-usb.ko 2024-04-16T05:17:56.840 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-16T05:17:56.840 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan.ko 2024-04-16T05:17:56.840 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-16T05:17:56.840 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-16T05:17:56.840 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-16T05:17:56.840 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mct_u232.ko 2024-04-16T05:17:56.840 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7720.ko 2024-04-16T05:17:56.840 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7840.ko 2024-04-16T05:17:56.840 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/navman.ko 2024-04-16T05:17:56.840 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/omninet.ko 2024-04-16T05:17:56.840 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/opticon.ko 2024-04-16T05:17:56.840 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/option.ko 2024-04-16T05:17:56.840 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/oti6858.ko 2024-04-16T05:17:56.841 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/pl2303.ko 2024-04-16T05:17:56.841 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcaux.ko 2024-04-16T05:17:56.841 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcserial.ko 2024-04-16T05:17:56.841 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/quatech2.ko 2024-04-16T05:17:56.841 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/safe_serial.ko 2024-04-16T05:17:56.841 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/sierra.ko 2024-04-16T05:17:56.841 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-16T05:17:56.841 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ssu100.ko 2024-04-16T05:17:56.841 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/symbolserial.ko 2024-04-16T05:17:56.841 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-16T05:17:56.841 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-16T05:17:56.841 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_debug.ko 2024-04-16T05:17:56.841 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-16T05:17:56.841 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/visor.ko 2024-04-16T05:17:56.842 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/whiteheat.ko 2024-04-16T05:17:56.842 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-16T05:17:56.842 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage 2024-04-16T05:17:56.842 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/uas.ko 2024-04-16T05:17:56.842 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-16T05:17:56.842 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-16T05:17:56.842 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-16T05:17:56.842 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-16T05:17:56.842 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-16T05:17:56.842 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-16T05:17:56.842 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-16T05:17:56.842 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-karma.ko 2024-04-16T05:17:56.842 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-16T05:17:56.843 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-16T05:17:56.843 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-16T05:17:56.843 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-16T05:17:56.843 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-16T05:17:56.843 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/usb-storage.ko 2024-04-16T05:17:56.843 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip 2024-04-16T05:17:56.843 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-16T05:17:56.843 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-16T05:17:56.843 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-16T05:17:56.843 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio 2024-04-16T05:17:56.843 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci 2024-04-16T05:17:56.843 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-16T05:17:56.843 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-16T05:17:56.844 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio.ko 2024-04-16T05:17:56.844 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-16T05:17:56.844 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost 2024-04-16T05:17:56.844 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost.ko 2024-04-16T05:17:56.844 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-16T05:17:56.844 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_net.ko 2024-04-16T05:17:56.844 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_scsi.ko 2024-04-16T05:17:56.844 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video 2024-04-16T05:17:56.844 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight 2024-04-16T05:17:56.844 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/apple_bl.ko 2024-04-16T05:17:56.844 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lcd.ko 2024-04-16T05:17:56.844 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-16T05:17:56.844 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio 2024-04-16T05:17:56.844 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio.ko 2024-04-16T05:17:56.844 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_balloon.ko 2024-04-16T05:17:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_mmio.ko 2024-04-16T05:17:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci.ko 2024-04-16T05:17:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-16T05:17:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-16T05:17:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_ring.ko 2024-04-16T05:17:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1 2024-04-16T05:17:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters 2024-04-16T05:17:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2482.ko 2024-04-16T05:17:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2490.ko 2024-04-16T05:17:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves 2024-04-16T05:17:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-16T05:17:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-16T05:17:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-16T05:17:56.845 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-16T05:17:56.846 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-16T05:17:56.846 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-16T05:17:56.846 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-16T05:17:56.846 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-16T05:17:56.846 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-16T05:17:56.846 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-16T05:17:56.846 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-16T05:17:56.846 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/wire.ko 2024-04-16T05:17:56.847 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog 2024-04-16T05:17:56.847 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-16T05:17:56.847 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-16T05:17:56.847 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-16T05:17:56.847 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/hpwdt.ko 2024-04-16T05:17:56.847 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/i6300esb.ko 2024-04-16T05:17:56.847 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-16T05:17:56.847 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-16T05:17:56.847 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ib700wdt.ko 2024-04-16T05:17:56.847 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ibmasr.ko 2024-04-16T05:17:56.847 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-16T05:17:56.847 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-16T05:17:56.847 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it87_wdt.ko 2024-04-16T05:17:56.847 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/machzwd.ko 2024-04-16T05:17:56.848 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/nv_tco.ko 2024-04-16T05:17:56.848 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-16T05:17:56.848 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-16T05:17:56.848 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-16T05:17:56.848 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-16T05:17:56.848 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/softdog.ko 2024-04-16T05:17:56.848 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-16T05:17:56.848 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/via_wdt.ko 2024-04-16T05:17:56.848 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-16T05:17:56.848 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-16T05:17:56.848 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-16T05:17:56.848 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wdt_pci.ko 2024-04-16T05:17:56.848 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/xen_wdt.ko 2024-04-16T05:17:56.849 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen 2024-04-16T05:17:56.849 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-16T05:17:56.849 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-evtchn.ko 2024-04-16T05:17:56.849 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntalloc.ko 2024-04-16T05:17:56.849 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntdev.ko 2024-04-16T05:17:56.849 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback 2024-04-16T05:17:56.849 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-16T05:17:56.849 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-privcmd.ko 2024-04-16T05:17:56.849 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs 2024-04-16T05:17:56.849 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-16T05:17:56.849 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs 2024-04-16T05:17:56.849 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p 2024-04-16T05:17:56.849 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p/9p.ko 2024-04-16T05:17:56.849 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs 2024-04-16T05:17:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs/affs.ko 2024-04-16T05:17:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs 2024-04-16T05:17:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs/befs.ko 2024-04-16T05:17:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/binfmt_misc.ko 2024-04-16T05:17:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs 2024-04-16T05:17:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs/btrfs.ko 2024-04-16T05:17:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles 2024-04-16T05:17:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles/cachefiles.ko 2024-04-16T05:17:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph 2024-04-16T05:17:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph/ceph.ko 2024-04-16T05:17:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda 2024-04-16T05:17:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda/coda.ko 2024-04-16T05:17:56.850 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs 2024-04-16T05:17:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs/cramfs.ko 2024-04-16T05:17:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm 2024-04-16T05:17:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm/dlm.ko 2024-04-16T05:17:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs 2024-04-16T05:17:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-16T05:17:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs 2024-04-16T05:17:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs/f2fs.ko 2024-04-16T05:17:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat 2024-04-16T05:17:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/fat.ko 2024-04-16T05:17:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/msdos.ko 2024-04-16T05:17:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/vfat.ko 2024-04-16T05:17:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse 2024-04-16T05:17:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/cuse.ko 2024-04-16T05:17:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/fuse.ko 2024-04-16T05:17:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2 2024-04-16T05:17:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2/gfs2.ko 2024-04-16T05:17:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs 2024-04-16T05:17:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs/hfs.ko 2024-04-16T05:17:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus 2024-04-16T05:17:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus/hfsplus.ko 2024-04-16T05:17:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs 2024-04-16T05:17:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs/isofs.ko 2024-04-16T05:17:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs 2024-04-16T05:17:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs/jfs.ko 2024-04-16T05:17:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd 2024-04-16T05:17:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd/lockd.ko 2024-04-16T05:17:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix 2024-04-16T05:17:56.852 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix/minix.ko 2024-04-16T05:17:56.853 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs 2024-04-16T05:17:56.853 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs/netfs.ko 2024-04-16T05:17:56.853 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs 2024-04-16T05:17:56.853 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout 2024-04-16T05:17:56.853 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-16T05:17:56.853 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout 2024-04-16T05:17:56.853 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-16T05:17:56.853 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout 2024-04-16T05:17:56.853 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-16T05:17:56.853 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfs.ko 2024-04-16T05:17:56.853 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv3.ko 2024-04-16T05:17:56.853 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv4.ko 2024-04-16T05:17:56.853 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common 2024-04-16T05:17:56.853 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/grace.ko 2024-04-16T05:17:56.853 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/nfs_acl.ko 2024-04-16T05:17:56.854 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd 2024-04-16T05:17:56.854 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd/nfsd.ko 2024-04-16T05:17:56.854 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2 2024-04-16T05:17:56.854 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2/nilfs2.ko 2024-04-16T05:17:56.854 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls 2024-04-16T05:17:56.854 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-celtic.ko 2024-04-16T05:17:56.854 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-centeuro.ko 2024-04-16T05:17:56.854 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-croatian.ko 2024-04-16T05:17:56.854 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-cyrillic.ko 2024-04-16T05:17:56.854 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-gaelic.ko 2024-04-16T05:17:56.854 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-greek.ko 2024-04-16T05:17:56.854 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-iceland.ko 2024-04-16T05:17:56.854 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-inuit.ko 2024-04-16T05:17:56.855 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-roman.ko 2024-04-16T05:17:56.855 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-romanian.ko 2024-04-16T05:17:56.855 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-turkish.ko 2024-04-16T05:17:56.855 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1250.ko 2024-04-16T05:17:56.855 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1251.ko 2024-04-16T05:17:56.855 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1255.ko 2024-04-16T05:17:56.855 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp737.ko 2024-04-16T05:17:56.855 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp775.ko 2024-04-16T05:17:56.855 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp850.ko 2024-04-16T05:17:56.855 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp852.ko 2024-04-16T05:17:56.855 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp855.ko 2024-04-16T05:17:56.855 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp857.ko 2024-04-16T05:17:56.855 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp860.ko 2024-04-16T05:17:56.855 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp861.ko 2024-04-16T05:17:56.856 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp862.ko 2024-04-16T05:17:56.856 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp863.ko 2024-04-16T05:17:56.856 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp864.ko 2024-04-16T05:17:56.856 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp865.ko 2024-04-16T05:17:56.856 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp866.ko 2024-04-16T05:17:56.856 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp869.ko 2024-04-16T05:17:56.856 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp874.ko 2024-04-16T05:17:56.856 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp932.ko 2024-04-16T05:17:56.856 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp936.ko 2024-04-16T05:17:56.856 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp949.ko 2024-04-16T05:17:56.856 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp950.ko 2024-04-16T05:17:56.856 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_euc-jp.ko 2024-04-16T05:17:56.856 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-1.ko 2024-04-16T05:17:56.856 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-13.ko 2024-04-16T05:17:56.857 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-14.ko 2024-04-16T05:17:56.857 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-15.ko 2024-04-16T05:17:56.857 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-2.ko 2024-04-16T05:17:56.857 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-3.ko 2024-04-16T05:17:56.857 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-4.ko 2024-04-16T05:17:56.857 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-5.ko 2024-04-16T05:17:56.857 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-6.ko 2024-04-16T05:17:56.857 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-7.ko 2024-04-16T05:17:56.857 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-9.ko 2024-04-16T05:17:56.857 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-r.ko 2024-04-16T05:17:56.857 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-ru.ko 2024-04-16T05:17:56.857 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-u.ko 2024-04-16T05:17:56.857 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-16T05:17:56.857 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_utf8.ko 2024-04-16T05:17:56.858 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2 2024-04-16T05:17:56.858 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster 2024-04-16T05:17:56.858 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-16T05:17:56.858 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm 2024-04-16T05:17:56.858 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-16T05:17:56.858 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs 2024-04-16T05:17:56.858 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-16T05:17:56.858 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2.ko 2024-04-16T05:17:56.858 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-16T05:17:56.858 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-16T05:17:56.858 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-16T05:17:56.858 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs 2024-04-16T05:17:56.858 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs/overlay.ko 2024-04-16T05:17:56.858 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore 2024-04-16T05:17:56.859 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore/ramoops.ko 2024-04-16T05:17:56.859 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs 2024-04-16T05:17:56.859 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs/reiserfs.ko 2024-04-16T05:17:56.859 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs 2024-04-16T05:17:56.859 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs/romfs.ko 2024-04-16T05:17:56.859 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb 2024-04-16T05:17:56.859 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client 2024-04-16T05:17:56.859 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client/cifs.ko 2024-04-16T05:17:56.859 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common 2024-04-16T05:17:56.859 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_arc4.ko 2024-04-16T05:17:56.859 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_md4.ko 2024-04-16T05:17:56.859 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs 2024-04-16T05:17:56.859 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs/squashfs.ko 2024-04-16T05:17:56.859 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv 2024-04-16T05:17:56.859 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv/sysv.ko 2024-04-16T05:17:56.860 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf 2024-04-16T05:17:56.860 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf/udf.ko 2024-04-16T05:17:56.860 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs 2024-04-16T05:17:56.860 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs/ufs.ko 2024-04-16T05:17:56.860 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs 2024-04-16T05:17:56.860 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs/xfs.ko 2024-04-16T05:17:56.860 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel 2024-04-16T05:17:56.860 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu 2024-04-16T05:17:56.860 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu/rcutorture.ko 2024-04-16T05:17:56.860 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/torture.ko 2024-04-16T05:17:56.860 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace 2024-04-16T05:17:56.860 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-16T05:17:56.860 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib 2024-04-16T05:17:56.860 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/asn1_encoder.ko 2024-04-16T05:17:56.861 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc-itu-t.ko 2024-04-16T05:17:56.861 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc8.ko 2024-04-16T05:17:56.861 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto 2024-04-16T05:17:56.861 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libarc4.ko 2024-04-16T05:17:56.861 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libdes.ko 2024-04-16T05:17:56.861 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/libcrc32c.ko 2024-04-16T05:17:56.861 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lru_cache.ko 2024-04-16T05:17:56.861 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4 2024-04-16T05:17:56.861 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4_compress.ko 2024-04-16T05:17:56.861 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4hc_compress.ko 2024-04-16T05:17:56.861 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math 2024-04-16T05:17:56.862 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math/cordic.ko 2024-04-16T05:17:56.862 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6 2024-04-16T05:17:56.862 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6/raid6_pq.ko 2024-04-16T05:17:56.862 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon 2024-04-16T05:17:56.862 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-16T05:17:56.862 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_bm.ko 2024-04-16T05:17:56.862 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_fsm.ko 2024-04-16T05:17:56.862 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_kmp.ko 2024-04-16T05:17:56.862 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd 2024-04-16T05:17:56.862 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd/zstd_compress.ko 2024-04-16T05:17:56.862 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm 2024-04-16T05:17:56.862 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm/hwpoison-inject.ko 2024-04-16T05:17:56.862 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net 2024-04-16T05:17:56.863 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan 2024-04-16T05:17:56.863 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/6lowpan.ko 2024-04-16T05:17:56.863 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_dest.ko 2024-04-16T05:17:56.863 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_fragment.ko 2024-04-16T05:17:56.863 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_hop.ko 2024-04-16T05:17:56.863 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-16T05:17:56.863 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_mobility.ko 2024-04-16T05:17:56.863 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_routing.ko 2024-04-16T05:17:56.863 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_udp.ko 2024-04-16T05:17:56.863 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802 2024-04-16T05:17:56.863 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/garp.ko 2024-04-16T05:17:56.863 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/mrp.ko 2024-04-16T05:17:56.863 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/p8022.ko 2024-04-16T05:17:56.864 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/psnap.ko 2024-04-16T05:17:56.864 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/stp.ko 2024-04-16T05:17:56.864 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q 2024-04-16T05:17:56.864 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q/8021q.ko 2024-04-16T05:17:56.864 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p 2024-04-16T05:17:56.864 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet.ko 2024-04-16T05:17:56.864 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_fd.ko 2024-04-16T05:17:56.864 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_rdma.ko 2024-04-16T05:17:56.864 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_virtio.ko 2024-04-16T05:17:56.864 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv 2024-04-16T05:17:56.864 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv/batman-adv.ko 2024-04-16T05:17:56.864 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge 2024-04-16T05:17:56.864 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/br_netfilter.ko 2024-04-16T05:17:56.864 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/bridge.ko 2024-04-16T05:17:56.865 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter 2024-04-16T05:17:56.865 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-16T05:17:56.865 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-16T05:17:56.865 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-16T05:17:56.865 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-16T05:17:56.865 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-16T05:17:56.865 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-16T05:17:56.865 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-16T05:17:56.865 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-16T05:17:56.865 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-16T05:17:56.865 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-16T05:17:56.865 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-16T05:17:56.865 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-16T05:17:56.865 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-16T05:17:56.866 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-16T05:17:56.866 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-16T05:17:56.866 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-16T05:17:56.866 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-16T05:17:56.866 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-16T05:17:56.866 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-16T05:17:56.866 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-16T05:17:56.867 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtables.ko 2024-04-16T05:17:56.867 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-16T05:17:56.867 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-16T05:17:56.867 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-16T05:17:56.867 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph 2024-04-16T05:17:56.867 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph/libceph.ko 2024-04-16T05:17:56.867 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core 2024-04-16T05:17:56.867 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/failover.ko 2024-04-16T05:17:56.867 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/pktgen.ko 2024-04-16T05:17:56.867 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp 2024-04-16T05:17:56.867 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp.ko 2024-04-16T05:17:56.867 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_diag.ko 2024-04-16T05:17:56.867 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv4.ko 2024-04-16T05:17:56.868 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv6.ko 2024-04-16T05:17:56.868 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver 2024-04-16T05:17:56.868 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver/dns_resolver.ko 2024-04-16T05:17:56.868 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa 2024-04-16T05:17:56.868 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/dsa_core.ko 2024-04-16T05:17:56.868 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_brcm.ko 2024-04-16T05:17:56.868 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_dsa.ko 2024-04-16T05:17:56.868 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_none.ko 2024-04-16T05:17:56.868 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_trailer.ko 2024-04-16T05:17:56.868 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154 2024-04-16T05:17:56.868 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan 2024-04-16T05:17:56.868 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-16T05:17:56.868 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154.ko 2024-04-16T05:17:56.869 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-16T05:17:56.869 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4 2024-04-16T05:17:56.869 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ah4.ko 2024-04-16T05:17:56.869 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/esp4.ko 2024-04-16T05:17:56.869 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/fou.ko 2024-04-16T05:17:56.869 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/gre.ko 2024-04-16T05:17:56.869 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/inet_diag.ko 2024-04-16T05:17:56.869 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_gre.ko 2024-04-16T05:17:56.869 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_tunnel.ko 2024-04-16T05:17:56.869 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_vti.ko 2024-04-16T05:17:56.869 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipcomp.ko 2024-04-16T05:17:56.870 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipip.ko 2024-04-16T05:17:56.870 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter 2024-04-16T05:17:56.870 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-16T05:17:56.871 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-16T05:17:56.871 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-16T05:17:56.871 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-16T05:17:56.871 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-16T05:17:56.871 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-16T05:17:56.871 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-16T05:17:56.871 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-16T05:17:56.871 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-16T05:17:56.871 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-16T05:17:56.871 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-16T05:17:56.871 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-16T05:17:56.871 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-16T05:17:56.871 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-16T05:17:56.871 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-16T05:17:56.872 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-16T05:17:56.872 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-16T05:17:56.872 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-16T05:17:56.872 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-16T05:17:56.872 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-16T05:17:56.872 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-16T05:17:56.872 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_bic.ko 2024-04-16T05:17:56.872 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_dctcp.ko 2024-04-16T05:17:56.872 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_diag.ko 2024-04-16T05:17:56.872 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_highspeed.ko 2024-04-16T05:17:56.872 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_htcp.ko 2024-04-16T05:17:56.872 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_hybla.ko 2024-04-16T05:17:56.872 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_illinois.ko 2024-04-16T05:17:56.873 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_lp.ko 2024-04-16T05:17:56.873 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_scalable.ko 2024-04-16T05:17:56.873 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_vegas.ko 2024-04-16T05:17:56.873 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_veno.ko 2024-04-16T05:17:56.873 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_westwood.ko 2024-04-16T05:17:56.873 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_yeah.ko 2024-04-16T05:17:56.873 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tunnel4.ko 2024-04-16T05:17:56.873 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_diag.ko 2024-04-16T05:17:56.873 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_tunnel.ko 2024-04-16T05:17:56.873 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-16T05:17:56.873 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6 2024-04-16T05:17:56.873 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ah6.ko 2024-04-16T05:17:56.873 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/esp6.ko 2024-04-16T05:17:56.873 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/fou6.ko 2024-04-16T05:17:56.874 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_tunnel.ko 2024-04-16T05:17:56.874 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-16T05:17:56.874 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_vti.ko 2024-04-16T05:17:56.874 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ipcomp6.ko 2024-04-16T05:17:56.874 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter 2024-04-16T05:17:56.874 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-16T05:17:56.874 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-16T05:17:56.874 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-16T05:17:56.874 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-16T05:17:56.874 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-16T05:17:56.874 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-16T05:17:56.874 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-16T05:17:56.874 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-16T05:17:56.874 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-16T05:17:56.875 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-16T05:17:56.875 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-16T05:17:56.875 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-16T05:17:56.875 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-16T05:17:56.875 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-16T05:17:56.875 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-16T05:17:56.875 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-16T05:17:56.875 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-16T05:17:56.875 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-16T05:17:56.875 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-16T05:17:56.875 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-16T05:17:56.875 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-16T05:17:56.875 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-16T05:17:56.875 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-16T05:17:56.875 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-16T05:17:56.876 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/sit.ko 2024-04-16T05:17:56.876 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/tunnel6.ko 2024-04-16T05:17:56.876 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-16T05:17:56.876 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key 2024-04-16T05:17:56.876 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key/af_key.ko 2024-04-16T05:17:56.876 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp 2024-04-16T05:17:56.876 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_core.ko 2024-04-16T05:17:56.876 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-16T05:17:56.876 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_eth.ko 2024-04-16T05:17:56.876 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip.ko 2024-04-16T05:17:56.876 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip6.ko 2024-04-16T05:17:56.876 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_netlink.ko 2024-04-16T05:17:56.876 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ppp.ko 2024-04-16T05:17:56.876 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc 2024-04-16T05:17:56.877 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/llc.ko 2024-04-16T05:17:56.877 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154 2024-04-16T05:17:56.877 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154/mac802154.ko 2024-04-16T05:17:56.877 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls 2024-04-16T05:17:56.877 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls/mpls_gso.ko 2024-04-16T05:17:56.877 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter 2024-04-16T05:17:56.877 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset 2024-04-16T05:17:56.877 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set.ko 2024-04-16T05:17:56.877 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-16T05:17:56.877 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-16T05:17:56.877 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-16T05:17:56.877 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-16T05:17:56.877 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-16T05:17:56.877 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-16T05:17:56.877 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-16T05:17:56.878 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-16T05:17:56.878 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-16T05:17:56.878 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-16T05:17:56.878 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-16T05:17:56.878 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-16T05:17:56.878 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-16T05:17:56.878 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-16T05:17:56.878 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-16T05:17:56.878 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs 2024-04-16T05:17:56.878 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-16T05:17:56.878 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-16T05:17:56.878 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-16T05:17:56.878 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-16T05:17:56.878 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-16T05:17:56.879 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-16T05:17:56.879 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-16T05:17:56.879 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-16T05:17:56.879 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-16T05:17:56.879 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-16T05:17:56.879 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-16T05:17:56.879 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-16T05:17:56.879 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-16T05:17:56.879 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-16T05:17:56.879 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conncount.ko 2024-04-16T05:17:56.879 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack.ko 2024-04-16T05:17:56.879 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-16T05:17:56.879 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-16T05:17:56.879 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-16T05:17:56.880 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-16T05:17:56.880 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-16T05:17:56.880 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-16T05:17:56.880 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-16T05:17:56.880 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-16T05:17:56.880 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-16T05:17:56.880 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-16T05:17:56.880 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-16T05:17:56.880 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-16T05:17:56.880 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-16T05:17:56.880 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table.ko 2024-04-16T05:17:56.881 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-16T05:17:56.881 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_log_syslog.ko 2024-04-16T05:17:56.881 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat.ko 2024-04-16T05:17:56.881 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-16T05:17:56.881 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-16T05:17:56.881 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_irc.ko 2024-04-16T05:17:56.881 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_sip.ko 2024-04-16T05:17:56.881 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-16T05:17:56.881 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-16T05:17:56.881 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_tables.ko 2024-04-16T05:17:56.881 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink.ko 2024-04-16T05:17:56.881 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-16T05:17:56.881 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_log.ko 2024-04-16T05:17:56.881 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-16T05:17:56.882 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-16T05:17:56.882 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_chain_nat.ko 2024-04-16T05:17:56.882 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_compat.ko 2024-04-16T05:17:56.882 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_ct.ko 2024-04-16T05:17:56.882 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-16T05:17:56.882 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fib.ko 2024-04-16T05:17:56.882 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-16T05:17:56.882 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_hash.ko 2024-04-16T05:17:56.882 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_limit.ko 2024-04-16T05:17:56.882 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_log.ko 2024-04-16T05:17:56.882 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_masq.ko 2024-04-16T05:17:56.882 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_nat.ko 2024-04-16T05:17:56.882 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_numgen.ko 2024-04-16T05:17:56.882 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_queue.ko 2024-04-16T05:17:56.883 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_quota.ko 2024-04-16T05:17:56.883 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_redir.ko 2024-04-16T05:17:56.883 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject.ko 2024-04-16T05:17:56.883 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject_inet.ko 2024-04-16T05:17:56.883 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_socket.ko 2024-04-16T05:17:56.883 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_tunnel.ko 2024-04-16T05:17:56.883 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_AUDIT.ko 2024-04-16T05:17:56.883 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-16T05:17:56.883 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-16T05:17:56.883 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-16T05:17:56.883 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CT.ko 2024-04-16T05:17:56.884 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_DSCP.ko 2024-04-16T05:17:56.884 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HL.ko 2024-04-16T05:17:56.884 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HMARK.ko 2024-04-16T05:17:56.884 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-16T05:17:56.884 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LED.ko 2024-04-16T05:17:56.884 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LOG.ko 2024-04-16T05:17:56.884 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-16T05:17:56.884 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NETMAP.ko 2024-04-16T05:17:56.884 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFLOG.ko 2024-04-16T05:17:56.884 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-16T05:17:56.884 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_RATEEST.ko 2024-04-16T05:17:56.884 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-16T05:17:56.884 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_SECMARK.ko 2024-04-16T05:17:56.884 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-16T05:17:56.885 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-16T05:17:56.885 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TEE.ko 2024-04-16T05:17:56.885 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TPROXY.ko 2024-04-16T05:17:56.885 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TRACE.ko 2024-04-16T05:17:56.885 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_addrtype.ko 2024-04-16T05:17:56.885 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_bpf.ko 2024-04-16T05:17:56.885 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cgroup.ko 2024-04-16T05:17:56.885 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cluster.ko 2024-04-16T05:17:56.885 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_comment.ko 2024-04-16T05:17:56.885 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connbytes.ko 2024-04-16T05:17:56.885 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlabel.ko 2024-04-16T05:17:56.885 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlimit.ko 2024-04-16T05:17:56.885 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connmark.ko 2024-04-16T05:17:56.885 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_conntrack.ko 2024-04-16T05:17:56.885 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cpu.ko 2024-04-16T05:17:56.886 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dccp.ko 2024-04-16T05:17:56.886 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_devgroup.ko 2024-04-16T05:17:56.886 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dscp.ko 2024-04-16T05:17:56.886 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ecn.ko 2024-04-16T05:17:56.886 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_esp.ko 2024-04-16T05:17:56.886 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hashlimit.ko 2024-04-16T05:17:56.886 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_helper.ko 2024-04-16T05:17:56.886 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hl.ko 2024-04-16T05:17:56.886 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipcomp.ko 2024-04-16T05:17:56.886 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_iprange.ko 2024-04-16T05:17:56.886 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipvs.ko 2024-04-16T05:17:56.886 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_l2tp.ko 2024-04-16T05:17:56.886 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_length.ko 2024-04-16T05:17:56.886 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_limit.ko 2024-04-16T05:17:56.887 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mac.ko 2024-04-16T05:17:56.887 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mark.ko 2024-04-16T05:17:56.887 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_multiport.ko 2024-04-16T05:17:56.887 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nat.ko 2024-04-16T05:17:56.887 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nfacct.ko 2024-04-16T05:17:56.887 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_osf.ko 2024-04-16T05:17:56.887 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_owner.ko 2024-04-16T05:17:56.887 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_physdev.ko 2024-04-16T05:17:56.887 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_pkttype.ko 2024-04-16T05:17:56.887 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_policy.ko 2024-04-16T05:17:56.887 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_quota.ko 2024-04-16T05:17:56.887 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_rateest.ko 2024-04-16T05:17:56.887 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_realm.ko 2024-04-16T05:17:56.887 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_recent.ko 2024-04-16T05:17:56.888 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_sctp.ko 2024-04-16T05:17:56.888 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_set.ko 2024-04-16T05:17:56.888 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_socket.ko 2024-04-16T05:17:56.888 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_state.ko 2024-04-16T05:17:56.889 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_statistic.ko 2024-04-16T05:17:56.889 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_string.ko 2024-04-16T05:17:56.889 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_tcpmss.ko 2024-04-16T05:17:56.890 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_time.ko 2024-04-16T05:17:56.890 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_u32.ko 2024-04-16T05:17:56.890 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink 2024-04-16T05:17:56.890 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink/netlink_diag.ko 2024-04-16T05:17:56.890 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh 2024-04-16T05:17:56.890 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh/nsh.ko 2024-04-16T05:17:56.890 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch 2024-04-16T05:17:56.890 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/openvswitch.ko 2024-04-16T05:17:56.890 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-geneve.ko 2024-04-16T05:17:56.890 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-gre.ko 2024-04-16T05:17:56.890 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-vxlan.ko 2024-04-16T05:17:56.890 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet 2024-04-16T05:17:56.890 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet/af_packet_diag.ko 2024-04-16T05:17:56.890 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds 2024-04-16T05:17:56.891 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds.ko 2024-04-16T05:17:56.891 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_rdma.ko 2024-04-16T05:17:56.891 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_tcp.ko 2024-04-16T05:17:56.891 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched 2024-04-16T05:17:56.891 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_csum.ko 2024-04-16T05:17:56.891 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_gact.ko 2024-04-16T05:17:56.891 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_mirred.ko 2024-04-16T05:17:56.891 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_nat.ko 2024-04-16T05:17:56.891 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_pedit.ko 2024-04-16T05:17:56.891 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_police.ko 2024-04-16T05:17:56.891 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_simple.ko 2024-04-16T05:17:56.891 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_skbedit.ko 2024-04-16T05:17:56.891 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_vlan.ko 2024-04-16T05:17:56.891 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_basic.ko 2024-04-16T05:17:56.892 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_bpf.ko 2024-04-16T05:17:56.892 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_flow.ko 2024-04-16T05:17:56.892 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_fw.ko 2024-04-16T05:17:56.892 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_route.ko 2024-04-16T05:17:56.892 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_u32.ko 2024-04-16T05:17:56.892 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_cmp.ko 2024-04-16T05:17:56.892 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_ipset.ko 2024-04-16T05:17:56.892 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_meta.ko 2024-04-16T05:17:56.892 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_nbyte.ko 2024-04-16T05:17:56.892 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_text.ko 2024-04-16T05:17:56.892 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_u32.ko 2024-04-16T05:17:56.892 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_choke.ko 2024-04-16T05:17:56.892 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_codel.ko 2024-04-16T05:17:56.892 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_drr.ko 2024-04-16T05:17:56.893 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_fq.ko 2024-04-16T05:17:56.893 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_gred.ko 2024-04-16T05:17:56.893 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hfsc.ko 2024-04-16T05:17:56.893 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hhf.ko 2024-04-16T05:17:56.893 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_htb.ko 2024-04-16T05:17:56.893 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_ingress.ko 2024-04-16T05:17:56.893 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio.ko 2024-04-16T05:17:56.893 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio_lib.ko 2024-04-16T05:17:56.893 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_multiq.ko 2024-04-16T05:17:56.893 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_netem.ko 2024-04-16T05:17:56.893 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_pie.ko 2024-04-16T05:17:56.893 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_plug.ko 2024-04-16T05:17:56.893 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_prio.ko 2024-04-16T05:17:56.893 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_qfq.ko 2024-04-16T05:17:56.893 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_red.ko 2024-04-16T05:17:56.894 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfb.ko 2024-04-16T05:17:56.894 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfq.ko 2024-04-16T05:17:56.894 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_tbf.ko 2024-04-16T05:17:56.894 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_teql.ko 2024-04-16T05:17:56.894 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp 2024-04-16T05:17:56.894 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp.ko 2024-04-16T05:17:56.894 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp_diag.ko 2024-04-16T05:17:56.894 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc 2024-04-16T05:17:56.894 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss 2024-04-16T05:17:56.894 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-16T05:17:56.894 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-16T05:17:56.894 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/sunrpc.ko 2024-04-16T05:17:56.894 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma 2024-04-16T05:17:56.894 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-16T05:17:56.895 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc 2024-04-16T05:17:56.895 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc.ko 2024-04-16T05:17:56.895 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc_diag.ko 2024-04-16T05:17:56.895 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix 2024-04-16T05:17:56.895 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix/unix_diag.ko 2024-04-16T05:17:56.895 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock 2024-04-16T05:17:56.895 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-16T05:17:56.895 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-16T05:17:56.895 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock.ko 2024-04-16T05:17:56.895 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-16T05:17:56.895 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-16T05:17:56.895 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm 2024-04-16T05:17:56.895 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-16T05:17:56.895 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security 2024-04-16T05:17:56.895 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys 2024-04-16T05:17:56.896 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys 2024-04-16T05:17:56.896 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-16T05:17:56.896 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys 2024-04-16T05:17:56.896 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys/trusted.ko 2024-04-16T05:17:56.896 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias 2024-04-16T05:17:56.896 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias.bin 2024-04-16T05:17:56.896 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin 2024-04-16T05:17:56.896 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.alias.bin 2024-04-16T05:17:56.896 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.bin 2024-04-16T05:17:56.896 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.modinfo 2024-04-16T05:17:56.896 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep 2024-04-16T05:17:56.896 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep.bin 2024-04-16T05:17:56.896 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.devname 2024-04-16T05:17:56.896 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.order 2024-04-16T05:17:56.897 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.softdep 2024-04-16T05:17:56.897 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols 2024-04-16T05:17:56.897 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols.bin 2024-04-16T05:17:56.897 INFO:teuthology.orchestra.run.smithi027.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-16T05:17:56.901 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T05:17:56.901 DEBUG:teuthology.orchestra.run.smithi027:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-16T05:17:56.927 INFO:teuthology.orchestra.run.smithi027.stdout:postinstall scriptlet (using /bin/sh): 2024-04-16T05:17:56.927 INFO:teuthology.orchestra.run.smithi027.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-16T05:17:56.927 INFO:teuthology.orchestra.run.smithi027.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g3f60ed14f7c1 /lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-16T05:17:56.927 INFO:teuthology.orchestra.run.smithi027.stdout:fi 2024-04-16T05:17:56.927 INFO:teuthology.orchestra.run.smithi027.stdout:for file in vmlinuz System.map config; do 2024-04-16T05:17:56.927 INFO:teuthology.orchestra.run.smithi027.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1"; then 2024-04-16T05:17:56.927 INFO:teuthology.orchestra.run.smithi027.stdout: cp "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T05:17:56.927 INFO:teuthology.orchestra.run.smithi027.stdout: fi 2024-04-16T05:17:56.927 INFO:teuthology.orchestra.run.smithi027.stdout:done 2024-04-16T05:17:56.928 INFO:teuthology.orchestra.run.smithi027.stdout:if [ -d "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" ] && \ 2024-04-16T05:17:56.928 INFO:teuthology.orchestra.run.smithi027.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" >/dev/null 2>&1; then 2024-04-16T05:17:56.928 INFO:teuthology.orchestra.run.smithi027.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T05:17:56.928 INFO:teuthology.orchestra.run.smithi027.stdout: cp -r "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T05:17:56.928 INFO:teuthology.orchestra.run.smithi027.stdout:fi 2024-04-16T05:17:56.928 INFO:teuthology.orchestra.run.smithi027.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep" ]; then 2024-04-16T05:17:56.928 INFO:teuthology.orchestra.run.smithi027.stdout: /usr/sbin/depmod 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T05:17:56.928 INFO:teuthology.orchestra.run.smithi027.stdout:fi 2024-04-16T05:17:56.928 INFO:teuthology.orchestra.run.smithi027.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-16T05:17:56.928 INFO:teuthology.orchestra.run.smithi027.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-16T05:17:56.928 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-install remove 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T05:17:56.928 INFO:teuthology.orchestra.run.smithi027.stdout:fi 2024-04-16T05:17:56.929 DEBUG:teuthology.orchestra.run.smithi027:> sudo rpm -qi grub2-tools 2024-04-16T05:17:57.018 INFO:teuthology.orchestra.run.smithi027.stdout:Name : grub2-tools 2024-04-16T05:17:57.018 INFO:teuthology.orchestra.run.smithi027.stdout:Epoch : 1 2024-04-16T05:17:57.018 INFO:teuthology.orchestra.run.smithi027.stdout:Version : 2.06 2024-04-16T05:17:57.018 INFO:teuthology.orchestra.run.smithi027.stdout:Release : 68.el9 2024-04-16T05:17:57.018 INFO:teuthology.orchestra.run.smithi027.stdout:Architecture: x86_64 2024-04-16T05:17:57.018 INFO:teuthology.orchestra.run.smithi027.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T05:17:57.019 INFO:teuthology.orchestra.run.smithi027.stdout:Group : Unspecified 2024-04-16T05:17:57.019 INFO:teuthology.orchestra.run.smithi027.stdout:Size : 8264254 2024-04-16T05:17:57.019 INFO:teuthology.orchestra.run.smithi027.stdout:License : GPLv3+ 2024-04-16T05:17:57.019 INFO:teuthology.orchestra.run.smithi027.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T05:17:57.019 INFO:teuthology.orchestra.run.smithi027.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T05:17:57.019 INFO:teuthology.orchestra.run.smithi027.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T05:17:57.019 INFO:teuthology.orchestra.run.smithi027.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T05:17:57.019 INFO:teuthology.orchestra.run.smithi027.stdout:Packager : builder@centos.org 2024-04-16T05:17:57.019 INFO:teuthology.orchestra.run.smithi027.stdout:Vendor : CentOS 2024-04-16T05:17:57.019 INFO:teuthology.orchestra.run.smithi027.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T05:17:57.019 INFO:teuthology.orchestra.run.smithi027.stdout:Summary : Support tools for GRUB. 2024-04-16T05:17:57.019 INFO:teuthology.orchestra.run.smithi027.stdout:Description : 2024-04-16T05:17:57.019 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:17:57.019 INFO:teuthology.orchestra.run.smithi027.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T05:17:57.020 INFO:teuthology.orchestra.run.smithi027.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T05:17:57.020 INFO:teuthology.orchestra.run.smithi027.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T05:17:57.020 INFO:teuthology.orchestra.run.smithi027.stdout:hardware devices. 2024-04-16T05:17:57.020 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:17:57.020 INFO:teuthology.orchestra.run.smithi027.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T05:17:57.021 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T05:17:57.021 INFO:teuthology.task.kernel:Updating grub on smithi027 to boot 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T05:17:57.021 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T05:17:57.490 INFO:teuthology.orchestra.run.smithi027.stderr:Generating grub configuration file ... 2024-04-16T05:17:58.931 INFO:teuthology.orchestra.run.smithi027.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T05:17:58.968 INFO:teuthology.orchestra.run.smithi027.stderr:done 2024-04-16T05:17:58.971 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2024-04-16T05:17:58.987 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.4Awk2FjigR 2024-04-16T05:17:58.987 DEBUG:teuthology.orchestra.run.smithi027:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4Awk2FjigR 2024-04-16T05:17:59.085 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.4Awk2FjigR 2024-04-16T05:17:59.191 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.4Awk2FjigR is 6KB 2024-04-16T05:17:59.241 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.4Awk2FjigR 2024-04-16T05:17:59.256 DEBUG:teuthology.orchestra.run.smithi027:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T05:17:59.334 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T05:17:59.334 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T05:17:59.335 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T05:17:59.335 INFO:teuthology.orchestra.run.smithi027.stdout:15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1.conf 2024-04-16T05:17:59.336 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T05:17:59.495 DEBUG:teuthology.orchestra.run.smithi027:> sudo shutdown -r now 2024-04-16T05:18:01.848 INFO:teuthology.orchestra.run.smithi122.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T05:18:24.961 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi122.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-16T05:18:24.962 DEBUG:teuthology.orchestra.run.smithi122:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-16T05:18:25.036 INFO:teuthology.orchestra.run.smithi122.stdout:/boot/System.map-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T05:18:25.036 INFO:teuthology.orchestra.run.smithi122.stdout:/boot/config-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T05:18:25.036 INFO:teuthology.orchestra.run.smithi122.stdout:/boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T05:18:25.036 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/System.map 2024-04-16T05:18:25.036 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/config 2024-04-16T05:18:25.036 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel 2024-04-16T05:18:25.036 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch 2024-04-16T05:18:25.036 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86 2024-04-16T05:18:25.036 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto 2024-04-16T05:18:25.037 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-16T05:18:25.037 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-16T05:18:25.037 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-16T05:18:25.037 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-16T05:18:25.037 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-16T05:18:25.037 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-16T05:18:25.037 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-16T05:18:25.037 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-16T05:18:25.037 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-16T05:18:25.038 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-16T05:18:25.038 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-16T05:18:25.038 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-16T05:18:25.038 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-16T05:18:25.038 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-16T05:18:25.038 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-16T05:18:25.038 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-16T05:18:25.038 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-16T05:18:25.038 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-16T05:18:25.038 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-16T05:18:25.039 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-16T05:18:25.039 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel 2024-04-16T05:18:25.039 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu 2024-04-16T05:18:25.039 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce 2024-04-16T05:18:25.039 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-16T05:18:25.039 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm 2024-04-16T05:18:25.039 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-16T05:18:25.039 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-16T05:18:25.039 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm.ko 2024-04-16T05:18:25.039 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto 2024-04-16T05:18:25.039 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ansi_cprng.ko 2024-04-16T05:18:25.040 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/anubis.ko 2024-04-16T05:18:25.040 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/arc4.ko 2024-04-16T05:18:25.040 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx 2024-04-16T05:18:25.040 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_memcpy.ko 2024-04-16T05:18:25.040 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_pq.ko 2024-04-16T05:18:25.040 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-16T05:18:25.040 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_tx.ko 2024-04-16T05:18:25.040 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_xor.ko 2024-04-16T05:18:25.040 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/raid6test.ko 2024-04-16T05:18:25.040 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authenc.ko 2024-04-16T05:18:25.040 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authencesn.ko 2024-04-16T05:18:25.040 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blake2b_generic.ko 2024-04-16T05:18:25.040 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_common.ko 2024-04-16T05:18:25.041 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_generic.ko 2024-04-16T05:18:25.041 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/camellia_generic.ko 2024-04-16T05:18:25.041 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast5_generic.ko 2024-04-16T05:18:25.041 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast6_generic.ko 2024-04-16T05:18:25.041 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast_common.ko 2024-04-16T05:18:25.041 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ccm.ko 2024-04-16T05:18:25.041 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cmac.ko 2024-04-16T05:18:25.041 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crc32_generic.ko 2024-04-16T05:18:25.041 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_engine.ko 2024-04-16T05:18:25.041 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_user.ko 2024-04-16T05:18:25.041 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cts.ko 2024-04-16T05:18:25.041 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/des_generic.ko 2024-04-16T05:18:25.041 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/dh_generic.ko 2024-04-16T05:18:25.041 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/echainiv.ko 2024-04-16T05:18:25.042 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/essiv.ko 2024-04-16T05:18:25.042 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/fcrypt.ko 2024-04-16T05:18:25.042 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/khazad.ko 2024-04-16T05:18:25.042 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4.ko 2024-04-16T05:18:25.042 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4hc.ko 2024-04-16T05:18:25.042 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/md4.ko 2024-04-16T05:18:25.042 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/michael_mic.ko 2024-04-16T05:18:25.042 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcbc.ko 2024-04-16T05:18:25.042 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcrypt.ko 2024-04-16T05:18:25.042 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/rmd160.ko 2024-04-16T05:18:25.042 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/seed.ko 2024-04-16T05:18:25.042 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/serpent_generic.ko 2024-04-16T05:18:25.042 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tcrypt.ko 2024-04-16T05:18:25.042 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tea.ko 2024-04-16T05:18:25.043 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_common.ko 2024-04-16T05:18:25.043 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_generic.ko 2024-04-16T05:18:25.043 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/vmac.ko 2024-04-16T05:18:25.043 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/wp512.ko 2024-04-16T05:18:25.043 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xcbc.ko 2024-04-16T05:18:25.043 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xor.ko 2024-04-16T05:18:25.043 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xxhash_generic.ko 2024-04-16T05:18:25.043 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers 2024-04-16T05:18:25.043 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi 2024-04-16T05:18:25.043 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-16T05:18:25.043 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_pad.ko 2024-04-16T05:18:25.043 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/ec_sys.ko 2024-04-16T05:18:25.043 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/platform_profile.ko 2024-04-16T05:18:25.043 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbs.ko 2024-04-16T05:18:25.044 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbshc.ko 2024-04-16T05:18:25.044 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/video.ko 2024-04-16T05:18:25.044 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata 2024-04-16T05:18:25.044 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/acard-ahci.ko 2024-04-16T05:18:25.044 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ahci_platform.ko 2024-04-16T05:18:25.044 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ata_generic.ko 2024-04-16T05:18:25.044 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/libahci_platform.ko 2024-04-16T05:18:25.044 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_acpi.ko 2024-04-16T05:18:25.044 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ali.ko 2024-04-16T05:18:25.044 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_amd.ko 2024-04-16T05:18:25.044 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_artop.ko 2024-04-16T05:18:25.044 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atiixp.ko 2024-04-16T05:18:25.044 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atp867x.ko 2024-04-16T05:18:25.044 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd640.ko 2024-04-16T05:18:25.045 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd64x.ko 2024-04-16T05:18:25.045 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cypress.ko 2024-04-16T05:18:25.045 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_efar.ko 2024-04-16T05:18:25.045 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt366.ko 2024-04-16T05:18:25.045 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt37x.ko 2024-04-16T05:18:25.045 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-16T05:18:25.045 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-16T05:18:25.045 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it8213.ko 2024-04-16T05:18:25.045 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it821x.ko 2024-04-16T05:18:25.045 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_jmicron.ko 2024-04-16T05:18:25.045 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_marvell.ko 2024-04-16T05:18:25.045 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_mpiix.ko 2024-04-16T05:18:25.045 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_netcell.ko 2024-04-16T05:18:25.045 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ninja32.ko 2024-04-16T05:18:25.046 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87410.ko 2024-04-16T05:18:25.046 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87415.ko 2024-04-16T05:18:25.046 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_oldpiix.ko 2024-04-16T05:18:25.046 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_opti.ko 2024-04-16T05:18:25.046 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_optidma.ko 2024-04-16T05:18:25.046 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pcmcia.ko 2024-04-16T05:18:25.046 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-16T05:18:25.046 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-16T05:18:25.046 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_piccolo.ko 2024-04-16T05:18:25.046 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_rdc.ko 2024-04-16T05:18:25.046 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sch.ko 2024-04-16T05:18:25.046 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_serverworks.ko 2024-04-16T05:18:25.046 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sil680.ko 2024-04-16T05:18:25.047 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sis.ko 2024-04-16T05:18:25.047 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sl82c105.ko 2024-04-16T05:18:25.047 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_triflex.ko 2024-04-16T05:18:25.047 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_via.ko 2024-04-16T05:18:25.047 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pdc_adma.ko 2024-04-16T05:18:25.047 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_inic162x.ko 2024-04-16T05:18:25.047 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_mv.ko 2024-04-16T05:18:25.047 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_nv.ko 2024-04-16T05:18:25.048 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_promise.ko 2024-04-16T05:18:25.048 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_qstor.ko 2024-04-16T05:18:25.048 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil.ko 2024-04-16T05:18:25.048 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil24.ko 2024-04-16T05:18:25.048 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sis.ko 2024-04-16T05:18:25.048 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_svw.ko 2024-04-16T05:18:25.048 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sx4.ko 2024-04-16T05:18:25.048 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_uli.ko 2024-04-16T05:18:25.048 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_via.ko 2024-04-16T05:18:25.048 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_vsc.ko 2024-04-16T05:18:25.048 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay 2024-04-16T05:18:25.048 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay/ks0108.ko 2024-04-16T05:18:25.048 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base 2024-04-16T05:18:25.048 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap 2024-04-16T05:18:25.049 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-16T05:18:25.049 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-16T05:18:25.049 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma 2024-04-16T05:18:25.049 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma/bcma.ko 2024-04-16T05:18:25.049 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block 2024-04-16T05:18:25.049 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe 2024-04-16T05:18:25.049 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe/aoe.ko 2024-04-16T05:18:25.049 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/brd.ko 2024-04-16T05:18:25.049 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd 2024-04-16T05:18:25.049 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd/drbd.ko 2024-04-16T05:18:25.049 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/floppy.ko 2024-04-16T05:18:25.049 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/loop.ko 2024-04-16T05:18:25.049 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx 2024-04-16T05:18:25.049 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-16T05:18:25.050 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/nbd.ko 2024-04-16T05:18:25.050 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk 2024-04-16T05:18:25.050 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk/null_blk.ko 2024-04-16T05:18:25.050 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/pktcdvd.ko 2024-04-16T05:18:25.050 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/rbd.ko 2024-04-16T05:18:25.050 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/virtio_blk.ko 2024-04-16T05:18:25.050 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback 2024-04-16T05:18:25.050 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-16T05:18:25.050 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkfront.ko 2024-04-16T05:18:25.050 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram 2024-04-16T05:18:25.050 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram/zram.ko 2024-04-16T05:18:25.050 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char 2024-04-16T05:18:25.050 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hangcheck-timer.ko 2024-04-16T05:18:25.051 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random 2024-04-16T05:18:25.051 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-16T05:18:25.051 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-16T05:18:25.051 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-16T05:18:25.051 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/via-rng.ko 2024-04-16T05:18:25.051 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-16T05:18:25.051 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi 2024-04-16T05:18:25.051 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-16T05:18:25.051 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-16T05:18:25.051 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-16T05:18:25.051 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-16T05:18:25.051 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-16T05:18:25.051 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-16T05:18:25.051 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/lp.ko 2024-04-16T05:18:25.051 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave 2024-04-16T05:18:25.052 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave/mwave.ko 2024-04-16T05:18:25.052 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ppdev.ko 2024-04-16T05:18:25.052 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tlclk.ko 2024-04-16T05:18:25.052 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm 2024-04-16T05:18:25.052 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm.ko 2024-04-16T05:18:25.052 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-16T05:18:25.052 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-16T05:18:25.052 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-16T05:18:25.052 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-16T05:18:25.052 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-16T05:18:25.052 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/uv_mmtimer.ko 2024-04-16T05:18:25.052 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/virtio_console.ko 2024-04-16T05:18:25.052 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq 2024-04-16T05:18:25.052 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-16T05:18:25.053 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-16T05:18:25.053 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-16T05:18:25.053 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-16T05:18:25.053 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-16T05:18:25.053 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-16T05:18:25.053 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto 2024-04-16T05:18:25.053 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp 2024-04-16T05:18:25.053 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-16T05:18:25.053 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp.ko 2024-04-16T05:18:25.053 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel 2024-04-16T05:18:25.053 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat 2024-04-16T05:18:25.053 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common 2024-04-16T05:18:25.053 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-16T05:18:25.053 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-16T05:18:25.053 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-16T05:18:25.054 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-aes.ko 2024-04-16T05:18:25.054 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-sha.ko 2024-04-16T05:18:25.054 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio 2024-04-16T05:18:25.054 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-16T05:18:25.054 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca 2024-04-16T05:18:25.054 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca/dca.ko 2024-04-16T05:18:25.054 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq 2024-04-16T05:18:25.054 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-16T05:18:25.054 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma 2024-04-16T05:18:25.054 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw 2024-04-16T05:18:25.054 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-16T05:18:25.054 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat 2024-04-16T05:18:25.054 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-16T05:18:25.054 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac 2024-04-16T05:18:25.055 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/amd64_edac.ko 2024-04-16T05:18:25.055 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/e752x_edac.ko 2024-04-16T05:18:25.055 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/edac_mce_amd.ko 2024-04-16T05:18:25.055 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3000_edac.ko 2024-04-16T05:18:25.055 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3200_edac.ko 2024-04-16T05:18:25.055 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5100_edac.ko 2024-04-16T05:18:25.055 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5400_edac.ko 2024-04-16T05:18:25.055 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7300_edac.ko 2024-04-16T05:18:25.055 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7core_edac.ko 2024-04-16T05:18:25.055 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i82975x_edac.ko 2024-04-16T05:18:25.055 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/ie31200_edac.ko 2024-04-16T05:18:25.055 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/sb_edac.ko 2024-04-16T05:18:25.055 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/x38_edac.ko 2024-04-16T05:18:25.055 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware 2024-04-16T05:18:25.056 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/edd.ko 2024-04-16T05:18:25.056 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-16T05:18:25.056 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio 2024-04-16T05:18:25.056 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-ich.ko 2024-04-16T05:18:25.056 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-16T05:18:25.056 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid 2024-04-16T05:18:25.056 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-appleir.ko 2024-04-16T05:18:25.056 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-aureal.ko 2024-04-16T05:18:25.056 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-axff.ko 2024-04-16T05:18:25.056 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-dr.ko 2024-04-16T05:18:25.056 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elecom.ko 2024-04-16T05:18:25.056 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elo.ko 2024-04-16T05:18:25.056 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-emsff.ko 2024-04-16T05:18:25.056 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gaff.ko 2024-04-16T05:18:25.056 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gt683r.ko 2024-04-16T05:18:25.057 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gyration.ko 2024-04-16T05:18:25.057 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-16T05:18:25.057 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-16T05:18:25.057 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtekff.ko 2024-04-16T05:18:25.057 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-hyperv.ko 2024-04-16T05:18:25.057 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-icade.ko 2024-04-16T05:18:25.057 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-keytouch.ko 2024-04-16T05:18:25.057 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-kye.ko 2024-04-16T05:18:25.057 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lcpower.ko 2024-04-16T05:18:25.058 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-led.ko 2024-04-16T05:18:25.058 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lenovo.ko 2024-04-16T05:18:25.058 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-16T05:18:25.058 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-16T05:18:25.058 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-multitouch.ko 2024-04-16T05:18:25.058 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ortek.ko 2024-04-16T05:18:25.058 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-penmount.ko 2024-04-16T05:18:25.058 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-petalynx.ko 2024-04-16T05:18:25.058 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-picolcd.ko 2024-04-16T05:18:25.058 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-pl.ko 2024-04-16T05:18:25.058 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-plantronics.ko 2024-04-16T05:18:25.058 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-primax.ko 2024-04-16T05:18:25.058 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-rmi.ko 2024-04-16T05:18:25.059 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-16T05:18:25.059 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-common.ko 2024-04-16T05:18:25.059 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-16T05:18:25.059 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-16T05:18:25.059 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-16T05:18:25.059 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-16T05:18:25.059 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-16T05:18:25.059 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-16T05:18:25.059 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-16T05:18:25.059 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-16T05:18:25.059 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-16T05:18:25.059 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat.ko 2024-04-16T05:18:25.060 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-saitek.ko 2024-04-16T05:18:25.060 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-samsung.ko 2024-04-16T05:18:25.060 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-16T05:18:25.060 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sjoy.ko 2024-04-16T05:18:25.060 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sony.ko 2024-04-16T05:18:25.060 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-speedlink.ko 2024-04-16T05:18:25.060 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-steelseries.ko 2024-04-16T05:18:25.060 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sunplus.ko 2024-04-16T05:18:25.060 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-16T05:18:25.060 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tivo.ko 2024-04-16T05:18:25.060 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tmff.ko 2024-04-16T05:18:25.060 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-topseed.ko 2024-04-16T05:18:25.060 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-twinhan.ko 2024-04-16T05:18:25.060 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-uclogic.ko 2024-04-16T05:18:25.061 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-waltop.ko 2024-04-16T05:18:25.061 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-wiimote.ko 2024-04-16T05:18:25.061 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-xinmo.ko 2024-04-16T05:18:25.061 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zpff.ko 2024-04-16T05:18:25.061 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zydacron.ko 2024-04-16T05:18:25.061 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/uhid.ko 2024-04-16T05:18:25.061 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/wacom.ko 2024-04-16T05:18:25.061 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv 2024-04-16T05:18:25.061 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_balloon.ko 2024-04-16T05:18:25.061 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_utils.ko 2024-04-16T05:18:25.061 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_vmbus.ko 2024-04-16T05:18:25.061 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon 2024-04-16T05:18:25.061 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru.ko 2024-04-16T05:18:25.061 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru3.ko 2024-04-16T05:18:25.062 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-16T05:18:25.062 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7414.ko 2024-04-16T05:18:25.062 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7418.ko 2024-04-16T05:18:25.062 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adc128d818.ko 2024-04-16T05:18:25.062 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1025.ko 2024-04-16T05:18:25.062 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1026.ko 2024-04-16T05:18:25.062 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1029.ko 2024-04-16T05:18:25.062 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1031.ko 2024-04-16T05:18:25.062 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm9240.ko 2024-04-16T05:18:25.062 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ads7828.ko 2024-04-16T05:18:25.062 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7410.ko 2024-04-16T05:18:25.062 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7411.ko 2024-04-16T05:18:25.062 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7462.ko 2024-04-16T05:18:25.062 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7470.ko 2024-04-16T05:18:25.063 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7475.ko 2024-04-16T05:18:25.063 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7x10.ko 2024-04-16T05:18:25.063 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/amc6821.ko 2024-04-16T05:18:25.063 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/applesmc.ko 2024-04-16T05:18:25.063 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asb100.ko 2024-04-16T05:18:25.063 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asc7621.ko 2024-04-16T05:18:25.063 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-16T05:18:25.063 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/atxp1.ko 2024-04-16T05:18:25.063 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/coretemp.ko 2024-04-16T05:18:25.063 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-16T05:18:25.063 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dme1737.ko 2024-04-16T05:18:25.063 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds1621.ko 2024-04-16T05:18:25.063 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds620.ko 2024-04-16T05:18:25.063 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc1403.ko 2024-04-16T05:18:25.064 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc6w201.ko 2024-04-16T05:18:25.064 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71805f.ko 2024-04-16T05:18:25.064 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71882fg.ko 2024-04-16T05:18:25.064 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f75375s.ko 2024-04-16T05:18:25.064 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fam15h_power.ko 2024-04-16T05:18:25.064 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fschmd.ko 2024-04-16T05:18:25.064 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g760a.ko 2024-04-16T05:18:25.064 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g762.ko 2024-04-16T05:18:25.064 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl518sm.ko 2024-04-16T05:18:25.064 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl520sm.ko 2024-04-16T05:18:25.064 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-16T05:18:25.064 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5500_temp.ko 2024-04-16T05:18:25.064 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5k_amb.ko 2024-04-16T05:18:25.064 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmaem.ko 2024-04-16T05:18:25.065 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmpex.ko 2024-04-16T05:18:25.065 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina209.ko 2024-04-16T05:18:25.065 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina2xx.ko 2024-04-16T05:18:25.065 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/it87.ko 2024-04-16T05:18:25.065 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k10temp.ko 2024-04-16T05:18:25.065 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k8temp.ko 2024-04-16T05:18:25.065 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lineage-pem.ko 2024-04-16T05:18:25.065 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm63.ko 2024-04-16T05:18:25.065 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm73.ko 2024-04-16T05:18:25.065 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm75.ko 2024-04-16T05:18:25.065 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm77.ko 2024-04-16T05:18:25.065 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm78.ko 2024-04-16T05:18:25.065 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm80.ko 2024-04-16T05:18:25.065 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm83.ko 2024-04-16T05:18:25.065 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm85.ko 2024-04-16T05:18:25.066 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm87.ko 2024-04-16T05:18:25.066 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm90.ko 2024-04-16T05:18:25.066 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm92.ko 2024-04-16T05:18:25.066 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm93.ko 2024-04-16T05:18:25.066 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95234.ko 2024-04-16T05:18:25.066 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95241.ko 2024-04-16T05:18:25.066 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95245.ko 2024-04-16T05:18:25.066 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc2945.ko 2024-04-16T05:18:25.066 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4151.ko 2024-04-16T05:18:25.066 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4215.ko 2024-04-16T05:18:25.066 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4222.ko 2024-04-16T05:18:25.066 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4245.ko 2024-04-16T05:18:25.066 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4260.ko 2024-04-16T05:18:25.067 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4261.ko 2024-04-16T05:18:25.067 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max16065.ko 2024-04-16T05:18:25.067 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1619.ko 2024-04-16T05:18:25.067 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1668.ko 2024-04-16T05:18:25.067 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max197.ko 2024-04-16T05:18:25.067 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6639.ko 2024-04-16T05:18:25.067 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6650.ko 2024-04-16T05:18:25.067 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6697.ko 2024-04-16T05:18:25.067 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/mcp3021.ko 2024-04-16T05:18:25.067 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6683.ko 2024-04-16T05:18:25.067 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775-core.ko 2024-04-16T05:18:25.067 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775.ko 2024-04-16T05:18:25.067 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct7802.ko 2024-04-16T05:18:25.067 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87360.ko 2024-04-16T05:18:25.067 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87427.ko 2024-04-16T05:18:25.068 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pcf8591.ko 2024-04-16T05:18:25.068 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus 2024-04-16T05:18:25.068 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-16T05:18:25.068 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-16T05:18:25.068 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-16T05:18:25.068 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-16T05:18:25.068 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-16T05:18:25.068 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-16T05:18:25.068 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-16T05:18:25.068 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-16T05:18:25.068 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-16T05:18:25.068 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-16T05:18:25.068 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-16T05:18:25.068 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-16T05:18:25.069 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/powr1220.ko 2024-04-16T05:18:25.069 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5627.ko 2024-04-16T05:18:25.069 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5636.ko 2024-04-16T05:18:25.069 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-16T05:18:25.069 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht15.ko 2024-04-16T05:18:25.069 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht21.ko 2024-04-16T05:18:25.069 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/shtc1.ko 2024-04-16T05:18:25.069 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sis5595.ko 2024-04-16T05:18:25.069 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47b397.ko 2024-04-16T05:18:25.069 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m1.ko 2024-04-16T05:18:25.069 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m192.ko 2024-04-16T05:18:25.069 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/thmc50.ko 2024-04-16T05:18:25.069 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp102.ko 2024-04-16T05:18:25.070 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp103.ko 2024-04-16T05:18:25.070 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp401.ko 2024-04-16T05:18:25.070 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp421.ko 2024-04-16T05:18:25.070 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via-cputemp.ko 2024-04-16T05:18:25.070 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via686a.ko 2024-04-16T05:18:25.070 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt1211.ko 2024-04-16T05:18:25.070 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt8231.ko 2024-04-16T05:18:25.070 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627ehf.ko 2024-04-16T05:18:25.070 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627hf.ko 2024-04-16T05:18:25.070 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83781d.ko 2024-04-16T05:18:25.070 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83791d.ko 2024-04-16T05:18:25.070 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83792d.ko 2024-04-16T05:18:25.070 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83793.ko 2024-04-16T05:18:25.070 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83795.ko 2024-04-16T05:18:25.071 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l785ts.ko 2024-04-16T05:18:25.071 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l786ng.ko 2024-04-16T05:18:25.071 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c 2024-04-16T05:18:25.071 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos 2024-04-16T05:18:25.072 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-16T05:18:25.072 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-16T05:18:25.072 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses 2024-04-16T05:18:25.072 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-16T05:18:25.072 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-16T05:18:25.072 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-16T05:18:25.072 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-16T05:18:25.072 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-16T05:18:25.072 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-16T05:18:25.072 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-16T05:18:25.072 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-16T05:18:25.072 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-16T05:18:25.072 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-16T05:18:25.073 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-16T05:18:25.073 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-16T05:18:25.073 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-16T05:18:25.073 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-16T05:18:25.073 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-16T05:18:25.073 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-16T05:18:25.073 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-16T05:18:25.073 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-16T05:18:25.073 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-16T05:18:25.073 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-16T05:18:25.073 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-16T05:18:25.073 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-16T05:18:25.073 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-16T05:18:25.074 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-dev.ko 2024-04-16T05:18:25.074 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-mux.ko 2024-04-16T05:18:25.074 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-16T05:18:25.074 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-smbus.ko 2024-04-16T05:18:25.074 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-stub.ko 2024-04-16T05:18:25.074 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband 2024-04-16T05:18:25.074 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core 2024-04-16T05:18:25.074 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-16T05:18:25.074 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_core.ko 2024-04-16T05:18:25.074 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-16T05:18:25.074 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-16T05:18:25.074 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-16T05:18:25.074 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-16T05:18:25.075 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-16T05:18:25.075 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw 2024-04-16T05:18:25.075 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4 2024-04-16T05:18:25.075 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-16T05:18:25.075 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4 2024-04-16T05:18:25.075 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-16T05:18:25.075 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5 2024-04-16T05:18:25.075 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-16T05:18:25.075 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca 2024-04-16T05:18:25.075 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-16T05:18:25.075 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp 2024-04-16T05:18:25.075 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib 2024-04-16T05:18:25.075 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-16T05:18:25.076 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser 2024-04-16T05:18:25.076 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-16T05:18:25.076 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert 2024-04-16T05:18:25.076 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-16T05:18:25.076 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp 2024-04-16T05:18:25.076 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-16T05:18:25.076 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt 2024-04-16T05:18:25.076 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-16T05:18:25.076 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input 2024-04-16T05:18:25.076 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport 2024-04-16T05:18:25.076 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-16T05:18:25.076 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-16T05:18:25.076 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/gameport.ko 2024-04-16T05:18:25.076 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/joydev.ko 2024-04-16T05:18:25.077 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard 2024-04-16T05:18:25.077 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-16T05:18:25.077 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse 2024-04-16T05:18:25.077 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/appletouch.ko 2024-04-16T05:18:25.077 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/bcm5974.ko 2024-04-16T05:18:25.077 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/cyapatp.ko 2024-04-16T05:18:25.077 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-16T05:18:25.077 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/sermouse.ko 2024-04-16T05:18:25.077 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-16T05:18:25.077 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-16T05:18:25.077 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-16T05:18:25.077 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4 2024-04-16T05:18:25.077 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-16T05:18:25.077 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio 2024-04-16T05:18:25.078 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/altera_ps2.ko 2024-04-16T05:18:25.078 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/arc_ps2.ko 2024-04-16T05:18:25.078 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-16T05:18:25.078 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/serio_raw.ko 2024-04-16T05:18:25.078 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/sparse-keymap.ko 2024-04-16T05:18:25.078 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds 2024-04-16T05:18:25.078 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-blinkm.ko 2024-04-16T05:18:25.078 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3530.ko 2024-04-16T05:18:25.078 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lp3944.ko 2024-04-16T05:18:25.078 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-ss4200.ko 2024-04-16T05:18:25.078 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger 2024-04-16T05:18:25.078 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-16T05:18:25.078 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-16T05:18:25.078 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-16T05:18:25.079 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-16T05:18:25.079 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-16T05:18:25.079 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-16T05:18:25.079 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-16T05:18:25.079 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-16T05:18:25.079 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-16T05:18:25.079 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md 2024-04-16T05:18:25.079 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache 2024-04-16T05:18:25.079 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache/bcache.ko 2024-04-16T05:18:25.079 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-bio-prison.ko 2024-04-16T05:18:25.079 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache-smq.ko 2024-04-16T05:18:25.079 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache.ko 2024-04-16T05:18:25.079 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-crypt.ko 2024-04-16T05:18:25.079 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-delay.ko 2024-04-16T05:18:25.080 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-flakey.ko 2024-04-16T05:18:25.080 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-log-userspace.ko 2024-04-16T05:18:25.080 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-multipath.ko 2024-04-16T05:18:25.080 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-queue-length.ko 2024-04-16T05:18:25.080 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-raid.ko 2024-04-16T05:18:25.080 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-round-robin.ko 2024-04-16T05:18:25.080 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-service-time.ko 2024-04-16T05:18:25.080 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-switch.ko 2024-04-16T05:18:25.080 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-thin-pool.ko 2024-04-16T05:18:25.080 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-verity.ko 2024-04-16T05:18:25.080 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data 2024-04-16T05:18:25.080 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-16T05:18:25.080 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid0.ko 2024-04-16T05:18:25.080 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid1.ko 2024-04-16T05:18:25.081 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid10.ko 2024-04-16T05:18:25.081 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid456.ko 2024-04-16T05:18:25.081 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message 2024-04-16T05:18:25.081 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion 2024-04-16T05:18:25.081 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptbase.ko 2024-04-16T05:18:25.081 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptctl.ko 2024-04-16T05:18:25.081 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptfc.ko 2024-04-16T05:18:25.081 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptlan.ko 2024-04-16T05:18:25.081 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptsas.ko 2024-04-16T05:18:25.081 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptscsih.ko 2024-04-16T05:18:25.081 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptspi.ko 2024-04-16T05:18:25.081 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd 2024-04-16T05:18:25.081 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_ich.ko 2024-04-16T05:18:25.081 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_sch.ko 2024-04-16T05:18:25.082 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/mfd-core.ko 2024-04-16T05:18:25.082 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/sm501.ko 2024-04-16T05:18:25.082 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/viperboard.ko 2024-04-16T05:18:25.082 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/vx855.ko 2024-04-16T05:18:25.082 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/wl1273-core.ko 2024-04-16T05:18:25.082 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc 2024-04-16T05:18:25.082 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl 2024-04-16T05:18:25.082 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-16T05:18:25.082 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds9802als.ko 2024-04-16T05:18:25.082 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds990x.ko 2024-04-16T05:18:25.082 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/bh1770glc.ko 2024-04-16T05:18:25.082 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710 2024-04-16T05:18:25.082 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710/cb710.ko 2024-04-16T05:18:25.082 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom 2024-04-16T05:18:25.083 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/at24.ko 2024-04-16T05:18:25.083 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-16T05:18:25.083 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/max6875.ko 2024-04-16T05:18:25.083 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/enclosure.ko 2024-04-16T05:18:25.083 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/hpilo.ko 2024-04-16T05:18:25.083 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29003.ko 2024-04-16T05:18:25.083 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29020.ko 2024-04-16T05:18:25.083 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d 2024-04-16T05:18:25.083 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-16T05:18:25.083 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-16T05:18:25.083 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei 2024-04-16T05:18:25.083 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-me.ko 2024-04-16T05:18:25.083 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-txe.ko 2024-04-16T05:18:25.084 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei.ko 2024-04-16T05:18:25.084 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru 2024-04-16T05:18:25.084 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-16T05:18:25.084 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp 2024-04-16T05:18:25.084 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-16T05:18:25.084 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-16T05:18:25.084 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-16T05:18:25.084 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_7xx1.ko 2024-04-16T05:18:25.085 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_core.ko 2024-04-16T05:18:25.085 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tsl2550.ko 2024-04-16T05:18:25.085 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_balloon.ko 2024-04-16T05:18:25.085 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci 2024-04-16T05:18:25.085 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-16T05:18:25.085 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net 2024-04-16T05:18:25.085 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding 2024-04-16T05:18:25.085 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding/bonding.ko 2024-04-16T05:18:25.085 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa 2024-04-16T05:18:25.085 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53 2024-04-16T05:18:25.085 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-16T05:18:25.085 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-16T05:18:25.086 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-16T05:18:25.086 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx 2024-04-16T05:18:25.086 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-16T05:18:25.086 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dummy.ko 2024-04-16T05:18:25.086 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/eql.ko 2024-04-16T05:18:25.086 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet 2024-04-16T05:18:25.086 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com 2024-04-16T05:18:25.086 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-16T05:18:25.086 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-16T05:18:25.086 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-16T05:18:25.086 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-16T05:18:25.087 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390 2024-04-16T05:18:25.087 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-16T05:18:25.087 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-16T05:18:25.087 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-16T05:18:25.087 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-16T05:18:25.087 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec 2024-04-16T05:18:25.087 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-16T05:18:25.087 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere 2024-04-16T05:18:25.087 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-16T05:18:25.087 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon 2024-04-16T05:18:25.087 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-16T05:18:25.088 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera 2024-04-16T05:18:25.088 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-16T05:18:25.088 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd 2024-04-16T05:18:25.088 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-16T05:18:25.088 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-16T05:18:25.088 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-16T05:18:25.088 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros 2024-04-16T05:18:25.088 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx 2024-04-16T05:18:25.088 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-16T05:18:25.088 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-16T05:18:25.088 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-16T05:18:25.088 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-16T05:18:25.088 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-16T05:18:25.088 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx 2024-04-16T05:18:25.088 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-16T05:18:25.089 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-16T05:18:25.089 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom 2024-04-16T05:18:25.089 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-16T05:18:25.089 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-16T05:18:25.089 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-16T05:18:25.089 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-16T05:18:25.089 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-16T05:18:25.089 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet 2024-04-16T05:18:25.089 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-16T05:18:25.089 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-16T05:18:25.089 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade 2024-04-16T05:18:25.089 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna 2024-04-16T05:18:25.089 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-16T05:18:25.089 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio 2024-04-16T05:18:25.090 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-16T05:18:25.090 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-16T05:18:25.090 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-16T05:18:25.090 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-16T05:18:25.090 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-16T05:18:25.090 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-16T05:18:25.090 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-16T05:18:25.090 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-16T05:18:25.090 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-16T05:18:25.090 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-16T05:18:25.090 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco 2024-04-16T05:18:25.090 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic 2024-04-16T05:18:25.090 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-16T05:18:25.091 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec 2024-04-16T05:18:25.091 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip 2024-04-16T05:18:25.091 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-16T05:18:25.091 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-16T05:18:25.091 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-16T05:18:25.091 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-16T05:18:25.091 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-16T05:18:25.091 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-16T05:18:25.091 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink 2024-04-16T05:18:25.091 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-16T05:18:25.091 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-16T05:18:25.091 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dnet.ko 2024-04-16T05:18:25.091 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex 2024-04-16T05:18:25.091 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet 2024-04-16T05:18:25.092 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-16T05:18:25.092 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ethoc.ko 2024-04-16T05:18:25.092 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fealnx.ko 2024-04-16T05:18:25.092 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel 2024-04-16T05:18:25.092 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-16T05:18:25.092 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000 2024-04-16T05:18:25.092 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-16T05:18:25.092 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e 2024-04-16T05:18:25.092 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-16T05:18:25.092 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k 2024-04-16T05:18:25.092 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-16T05:18:25.092 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e 2024-04-16T05:18:25.093 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-16T05:18:25.093 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf 2024-04-16T05:18:25.093 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-16T05:18:25.093 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb 2024-04-16T05:18:25.093 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-16T05:18:25.093 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf 2024-04-16T05:18:25.093 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-16T05:18:25.093 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-16T05:18:25.093 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-16T05:18:25.093 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-16T05:18:25.093 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-16T05:18:25.093 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/jme.ko 2024-04-16T05:18:25.093 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell 2024-04-16T05:18:25.093 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-16T05:18:25.094 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-16T05:18:25.094 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-16T05:18:25.094 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox 2024-04-16T05:18:25.094 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-16T05:18:25.094 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-16T05:18:25.094 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-16T05:18:25.094 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-16T05:18:25.094 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-16T05:18:25.094 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-16T05:18:25.094 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel 2024-04-16T05:18:25.094 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-16T05:18:25.094 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom 2024-04-16T05:18:25.094 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-16T05:18:25.094 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-16T05:18:25.095 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi 2024-04-16T05:18:25.095 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-16T05:18:25.095 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-16T05:18:25.095 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion 2024-04-16T05:18:25.095 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-16T05:18:25.095 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia 2024-04-16T05:18:25.095 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-16T05:18:25.095 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines 2024-04-16T05:18:25.095 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-16T05:18:25.095 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-16T05:18:25.095 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic 2024-04-16T05:18:25.095 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-16T05:18:25.095 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-16T05:18:25.095 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-16T05:18:25.096 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-16T05:18:25.096 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-16T05:18:25.096 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc 2024-04-16T05:18:25.096 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-16T05:18:25.096 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek 2024-04-16T05:18:25.096 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-16T05:18:25.096 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-16T05:18:25.096 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-16T05:18:25.096 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-16T05:18:25.096 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker 2024-04-16T05:18:25.096 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-16T05:18:25.096 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc 2024-04-16T05:18:25.096 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-16T05:18:25.096 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan 2024-04-16T05:18:25.096 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-16T05:18:25.097 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis 2024-04-16T05:18:25.097 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-16T05:18:25.097 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-16T05:18:25.097 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc 2024-04-16T05:18:25.097 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-16T05:18:25.097 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-16T05:18:25.097 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-16T05:18:25.097 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-16T05:18:25.097 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro 2024-04-16T05:18:25.097 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-16T05:18:25.097 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-16T05:18:25.097 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-16T05:18:25.097 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun 2024-04-16T05:18:25.097 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-16T05:18:25.098 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-16T05:18:25.098 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-16T05:18:25.098 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-16T05:18:25.098 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti 2024-04-16T05:18:25.098 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-16T05:18:25.098 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti 2024-04-16T05:18:25.098 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-16T05:18:25.098 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via 2024-04-16T05:18:25.098 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-16T05:18:25.098 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-16T05:18:25.098 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet 2024-04-16T05:18:25.098 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-16T05:18:25.098 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-16T05:18:25.098 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom 2024-04-16T05:18:25.099 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-16T05:18:25.099 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/geneve.ko 2024-04-16T05:18:25.099 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv 2024-04-16T05:18:25.099 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-16T05:18:25.099 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154 2024-04-16T05:18:25.099 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-16T05:18:25.099 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ifb.ko 2024-04-16T05:18:25.099 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan 2024-04-16T05:18:25.099 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-16T05:18:25.099 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvlan.ko 2024-04-16T05:18:25.099 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvtap.ko 2024-04-16T05:18:25.099 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio 2024-04-16T05:18:25.099 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio.ko 2024-04-16T05:18:25.099 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-16T05:18:25.100 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-16T05:18:25.100 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-16T05:18:25.100 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mii.ko 2024-04-16T05:18:25.100 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/net_failover.ko 2024-04-16T05:18:25.100 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/netconsole.ko 2024-04-16T05:18:25.100 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/nlmon.ko 2024-04-16T05:18:25.100 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs 2024-04-16T05:18:25.100 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-16T05:18:25.100 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-16T05:18:25.100 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy 2024-04-16T05:18:25.100 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/amd.ko 2024-04-16T05:18:25.100 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-16T05:18:25.100 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-16T05:18:25.100 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-16T05:18:25.100 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm87xx.ko 2024-04-16T05:18:25.101 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/broadcom.ko 2024-04-16T05:18:25.101 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/cicada.ko 2024-04-16T05:18:25.101 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/davicom.ko 2024-04-16T05:18:25.101 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/dp83640.ko 2024-04-16T05:18:25.101 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/et1011c.ko 2024-04-16T05:18:25.101 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/icplus.ko 2024-04-16T05:18:25.101 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/lxt.ko 2024-04-16T05:18:25.101 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/marvell.ko 2024-04-16T05:18:25.101 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/micrel.ko 2024-04-16T05:18:25.101 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/national.ko 2024-04-16T05:18:25.101 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/phylink.ko 2024-04-16T05:18:25.101 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/qsemi.ko 2024-04-16T05:18:25.101 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/realtek.ko 2024-04-16T05:18:25.101 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/smsc.ko 2024-04-16T05:18:25.102 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/ste10Xp.ko 2024-04-16T05:18:25.102 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/vitesse.ko 2024-04-16T05:18:25.102 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp 2024-04-16T05:18:25.102 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-16T05:18:25.102 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_async.ko 2024-04-16T05:18:25.102 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-16T05:18:25.102 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-16T05:18:25.102 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-16T05:18:25.102 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-16T05:18:25.102 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppoe.ko 2024-04-16T05:18:25.102 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppox.ko 2024-04-16T05:18:25.102 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pptp.ko 2024-04-16T05:18:25.102 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip 2024-04-16T05:18:25.102 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slhc.ko 2024-04-16T05:18:25.103 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slip.ko 2024-04-16T05:18:25.103 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/sungem_phy.ko 2024-04-16T05:18:25.103 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tap.ko 2024-04-16T05:18:25.103 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team 2024-04-16T05:18:25.103 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team.ko 2024-04-16T05:18:25.103 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-16T05:18:25.103 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-16T05:18:25.103 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-16T05:18:25.103 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_random.ko 2024-04-16T05:18:25.103 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-16T05:18:25.103 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tun.ko 2024-04-16T05:18:25.103 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/veth.ko 2024-04-16T05:18:25.103 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/virtio_net.ko 2024-04-16T05:18:25.103 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3 2024-04-16T05:18:25.103 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-16T05:18:25.104 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan 2024-04-16T05:18:25.104 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan/vxlan.ko 2024-04-16T05:18:25.104 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback 2024-04-16T05:18:25.104 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-16T05:18:25.104 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netfront.ko 2024-04-16T05:18:25.104 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb 2024-04-16T05:18:25.104 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb/ntb.ko 2024-04-16T05:18:25.104 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme 2024-04-16T05:18:25.104 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host 2024-04-16T05:18:25.104 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme-core.ko 2024-04-16T05:18:25.104 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme.ko 2024-04-16T05:18:25.105 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport 2024-04-16T05:18:25.105 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport.ko 2024-04-16T05:18:25.105 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_cs.ko 2024-04-16T05:18:25.105 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_pc.ko 2024-04-16T05:18:25.105 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_serial.ko 2024-04-16T05:18:25.105 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci 2024-04-16T05:18:25.105 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug 2024-04-16T05:18:25.105 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-16T05:18:25.105 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie 2024-04-16T05:18:25.105 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-16T05:18:25.105 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/xen-pcifront.ko 2024-04-16T05:18:25.105 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia 2024-04-16T05:18:25.106 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/i82092.ko 2024-04-16T05:18:25.106 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pd6729.ko 2024-04-16T05:18:25.106 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-16T05:18:25.106 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl 2024-04-16T05:18:25.106 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel 2024-04-16T05:18:25.106 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-16T05:18:25.106 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform 2024-04-16T05:18:25.106 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86 2024-04-16T05:18:25.106 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-16T05:18:25.106 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acerhdf.ko 2024-04-16T05:18:25.106 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-16T05:18:25.107 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-16T05:18:25.107 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-16T05:18:25.107 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-16T05:18:25.107 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-16T05:18:25.107 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-16T05:18:25.107 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-16T05:18:25.107 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-16T05:18:25.107 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-16T05:18:25.107 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/hdaps.ko 2024-04-16T05:18:25.107 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel 2024-04-16T05:18:25.107 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-16T05:18:25.107 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel_ips.ko 2024-04-16T05:18:25.107 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-16T05:18:25.107 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-16T05:18:25.107 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-16T05:18:25.108 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-16T05:18:25.108 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-16T05:18:25.108 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-16T05:18:25.108 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-16T05:18:25.108 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-16T05:18:25.108 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-16T05:18:25.108 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi.ko 2024-04-16T05:18:25.108 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap 2024-04-16T05:18:25.108 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-16T05:18:25.108 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-16T05:18:25.108 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps 2024-04-16T05:18:25.108 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients 2024-04-16T05:18:25.108 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-16T05:18:25.108 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-16T05:18:25.109 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps_parport.ko 2024-04-16T05:18:25.109 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/pps_core.ko 2024-04-16T05:18:25.109 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp 2024-04-16T05:18:25.109 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp.ko 2024-04-16T05:18:25.109 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp_kvm.ko 2024-04-16T05:18:25.109 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras 2024-04-16T05:18:25.109 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd 2024-04-16T05:18:25.109 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl 2024-04-16T05:18:25.109 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-16T05:18:25.109 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/fmpm.ko 2024-04-16T05:18:25.109 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc 2024-04-16T05:18:25.109 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-16T05:18:25.109 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-16T05:18:25.109 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-16T05:18:25.109 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-16T05:18:25.110 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-16T05:18:25.110 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-16T05:18:25.110 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-16T05:18:25.110 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-16T05:18:25.110 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-16T05:18:25.110 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-16T05:18:25.110 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-em3027.ko 2024-04-16T05:18:25.110 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-16T05:18:25.110 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-16T05:18:25.110 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-16T05:18:25.110 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-16T05:18:25.110 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-16T05:18:25.110 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-16T05:18:25.110 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max6900.ko 2024-04-16T05:18:25.111 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-16T05:18:25.111 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-16T05:18:25.111 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-16T05:18:25.111 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-16T05:18:25.111 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-16T05:18:25.111 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-16T05:18:25.111 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-16T05:18:25.111 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-16T05:18:25.111 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-16T05:18:25.111 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-16T05:18:25.111 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-16T05:18:25.111 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-16T05:18:25.111 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-x1205.ko 2024-04-16T05:18:25.111 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi 2024-04-16T05:18:25.112 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-9xxx.ko 2024-04-16T05:18:25.112 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-sas.ko 2024-04-16T05:18:25.112 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-xxxx.ko 2024-04-16T05:18:25.112 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/BusLogic.ko 2024-04-16T05:18:25.112 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/a100u2w.ko 2024-04-16T05:18:25.112 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid 2024-04-16T05:18:25.112 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-16T05:18:25.112 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/advansys.ko 2024-04-16T05:18:25.112 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx 2024-04-16T05:18:25.112 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-16T05:18:25.112 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-16T05:18:25.112 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/am53c974.ko 2024-04-16T05:18:25.112 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr 2024-04-16T05:18:25.112 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-16T05:18:25.113 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/atp870u.ko 2024-04-16T05:18:25.113 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi 2024-04-16T05:18:25.113 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-16T05:18:25.113 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa 2024-04-16T05:18:25.113 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa/bfa.ko 2024-04-16T05:18:25.113 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc 2024-04-16T05:18:25.113 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-16T05:18:25.113 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i 2024-04-16T05:18:25.113 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-16T05:18:25.113 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ch.ko 2024-04-16T05:18:25.113 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor 2024-04-16T05:18:25.113 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-16T05:18:25.113 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi 2024-04-16T05:18:25.113 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-16T05:18:25.114 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-16T05:18:25.114 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-16T05:18:25.114 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-16T05:18:25.114 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-16T05:18:25.114 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dc395x.ko 2024-04-16T05:18:25.114 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler 2024-04-16T05:18:25.114 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-16T05:18:25.114 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-16T05:18:25.114 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-16T05:18:25.114 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-16T05:18:25.114 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dmx3191d.ko 2024-04-16T05:18:25.114 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r 2024-04-16T05:18:25.114 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-16T05:18:25.114 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esp_scsi.ko 2024-04-16T05:18:25.115 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe 2024-04-16T05:18:25.115 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-16T05:18:25.115 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-16T05:18:25.115 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic 2024-04-16T05:18:25.115 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic/fnic.ko 2024-04-16T05:18:25.115 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hpsa.ko 2024-04-16T05:18:25.115 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hptiop.ko 2024-04-16T05:18:25.115 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hv_storvsc.ko 2024-04-16T05:18:25.115 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/initio.ko 2024-04-16T05:18:25.115 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ipr.ko 2024-04-16T05:18:25.115 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ips.ko 2024-04-16T05:18:25.115 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci 2024-04-16T05:18:25.115 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci/isci.ko 2024-04-16T05:18:25.115 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-16T05:18:25.116 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-16T05:18:25.116 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc 2024-04-16T05:18:25.116 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc/libfc.ko 2024-04-16T05:18:25.116 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi.ko 2024-04-16T05:18:25.116 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-16T05:18:25.116 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas 2024-04-16T05:18:25.116 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas/libsas.ko 2024-04-16T05:18:25.116 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc 2024-04-16T05:18:25.116 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-16T05:18:25.116 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid 2024-04-16T05:18:25.116 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid.ko 2024-04-16T05:18:25.116 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-16T05:18:25.116 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-16T05:18:25.116 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-16T05:18:25.117 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas 2024-04-16T05:18:25.117 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-16T05:18:25.117 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas 2024-04-16T05:18:25.117 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-16T05:18:25.117 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvumi.ko 2024-04-16T05:18:25.117 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001 2024-04-16T05:18:25.117 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-16T05:18:25.117 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pmcraid.ko 2024-04-16T05:18:25.117 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla1280.ko 2024-04-16T05:18:25.117 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx 2024-04-16T05:18:25.117 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-16T05:18:25.117 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-16T05:18:25.117 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx 2024-04-16T05:18:25.117 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-16T05:18:25.117 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/raid_class.ko 2024-04-16T05:18:25.118 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_debug.ko 2024-04-16T05:18:25.118 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-16T05:18:25.118 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-16T05:18:25.118 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-16T05:18:25.118 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-16T05:18:25.118 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-16T05:18:25.118 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ses.ko 2024-04-16T05:18:25.118 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/st.ko 2024-04-16T05:18:25.118 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/stex.ko 2024-04-16T05:18:25.118 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2 2024-04-16T05:18:25.118 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-16T05:18:25.118 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/virtio_scsi.ko 2024-04-16T05:18:25.118 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-16T05:18:25.119 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/wd719x.ko 2024-04-16T05:18:25.119 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb 2024-04-16T05:18:25.119 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb/ssb.ko 2024-04-16T05:18:25.119 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target 2024-04-16T05:18:25.119 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi 2024-04-16T05:18:25.119 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-16T05:18:25.119 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback 2024-04-16T05:18:25.119 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-16T05:18:25.119 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_file.ko 2024-04-16T05:18:25.119 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_iblock.ko 2024-04-16T05:18:25.119 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_mod.ko 2024-04-16T05:18:25.119 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_pscsi.ko 2024-04-16T05:18:25.119 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_user.ko 2024-04-16T05:18:25.119 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc 2024-04-16T05:18:25.120 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-16T05:18:25.120 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal 2024-04-16T05:18:25.120 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel 2024-04-16T05:18:25.120 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal 2024-04-16T05:18:25.120 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-16T05:18:25.120 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-16T05:18:25.120 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-16T05:18:25.120 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-16T05:18:25.120 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-16T05:18:25.120 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-16T05:18:25.120 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-16T05:18:25.120 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-16T05:18:25.120 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-16T05:18:25.120 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-16T05:18:25.121 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-16T05:18:25.121 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-16T05:18:25.121 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-16T05:18:25.121 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-16T05:18:25.121 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-16T05:18:25.121 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-16T05:18:25.121 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-16T05:18:25.121 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-16T05:18:25.121 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty 2024-04-16T05:18:25.121 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless 2024-04-16T05:18:25.121 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-16T05:18:25.121 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_gsm.ko 2024-04-16T05:18:25.121 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_hdlc.ko 2024-04-16T05:18:25.121 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/nozomi.ko 2024-04-16T05:18:25.122 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial 2024-04-16T05:18:25.122 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250 2024-04-16T05:18:25.122 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-16T05:18:25.122 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/arc_uart.ko 2024-04-16T05:18:25.122 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm 2024-04-16T05:18:25.122 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-16T05:18:25.122 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs 2024-04-16T05:18:25.122 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core 2024-04-16T05:18:25.122 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-16T05:18:25.122 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host 2024-04-16T05:18:25.122 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-16T05:18:25.122 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio 2024-04-16T05:18:25.122 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio.ko 2024-04-16T05:18:25.123 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_aec.ko 2024-04-16T05:18:25.123 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_cif.ko 2024-04-16T05:18:25.123 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_pci_generic.ko 2024-04-16T05:18:25.123 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_sercos3.ko 2024-04-16T05:18:25.123 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb 2024-04-16T05:18:25.123 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class 2024-04-16T05:18:25.123 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-acm.ko 2024-04-16T05:18:25.123 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-16T05:18:25.123 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usblp.ko 2024-04-16T05:18:25.123 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usbtmc.ko 2024-04-16T05:18:25.123 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host 2024-04-16T05:18:25.123 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-16T05:18:25.123 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image 2024-04-16T05:18:25.123 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/mdc800.ko 2024-04-16T05:18:25.124 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/microtek.ko 2024-04-16T05:18:25.124 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc 2024-04-16T05:18:25.124 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/adutux.ko 2024-04-16T05:18:25.124 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/appledisplay.ko 2024-04-16T05:18:25.124 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi26.ko 2024-04-16T05:18:25.124 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi62.ko 2024-04-16T05:18:25.124 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ezusb.ko 2024-04-16T05:18:25.124 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/idmouse.ko 2024-04-16T05:18:25.124 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/iowarrior.ko 2024-04-16T05:18:25.124 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-16T05:18:25.124 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ldusb.ko 2024-04-16T05:18:25.124 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/legousbtower.ko 2024-04-16T05:18:25.124 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga 2024-04-16T05:18:25.124 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-16T05:18:25.124 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-16T05:18:25.125 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usb3503.ko 2024-04-16T05:18:25.125 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usblcd.ko 2024-04-16T05:18:25.125 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-16T05:18:25.125 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/uss720.ko 2024-04-16T05:18:25.125 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/yurex.ko 2024-04-16T05:18:25.125 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy 2024-04-16T05:18:25.125 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-generic.ko 2024-04-16T05:18:25.125 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial 2024-04-16T05:18:25.125 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/aircable.ko 2024-04-16T05:18:25.125 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ark3116.ko 2024-04-16T05:18:25.125 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-16T05:18:25.125 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ch341.ko 2024-04-16T05:18:25.125 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cp210x.ko 2024-04-16T05:18:25.126 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cyberjack.ko 2024-04-16T05:18:25.126 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-16T05:18:25.126 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-16T05:18:25.126 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/empeg.ko 2024-04-16T05:18:25.126 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-16T05:18:25.126 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-16T05:18:25.126 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-16T05:18:25.126 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_ti.ko 2024-04-16T05:18:25.126 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipaq.ko 2024-04-16T05:18:25.126 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipw.ko 2024-04-16T05:18:25.126 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ir-usb.ko 2024-04-16T05:18:25.126 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-16T05:18:25.126 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan.ko 2024-04-16T05:18:25.126 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-16T05:18:25.127 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-16T05:18:25.127 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-16T05:18:25.127 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mct_u232.ko 2024-04-16T05:18:25.127 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7720.ko 2024-04-16T05:18:25.127 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7840.ko 2024-04-16T05:18:25.127 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/navman.ko 2024-04-16T05:18:25.127 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/omninet.ko 2024-04-16T05:18:25.127 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/opticon.ko 2024-04-16T05:18:25.127 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/option.ko 2024-04-16T05:18:25.127 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/oti6858.ko 2024-04-16T05:18:25.127 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/pl2303.ko 2024-04-16T05:18:25.127 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcaux.ko 2024-04-16T05:18:25.127 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcserial.ko 2024-04-16T05:18:25.127 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/quatech2.ko 2024-04-16T05:18:25.127 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/safe_serial.ko 2024-04-16T05:18:25.128 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/sierra.ko 2024-04-16T05:18:25.128 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-16T05:18:25.128 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ssu100.ko 2024-04-16T05:18:25.128 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/symbolserial.ko 2024-04-16T05:18:25.128 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-16T05:18:25.128 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-16T05:18:25.128 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_debug.ko 2024-04-16T05:18:25.128 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-16T05:18:25.128 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/visor.ko 2024-04-16T05:18:25.128 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/whiteheat.ko 2024-04-16T05:18:25.128 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-16T05:18:25.128 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage 2024-04-16T05:18:25.128 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/uas.ko 2024-04-16T05:18:25.128 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-16T05:18:25.129 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-16T05:18:25.129 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-16T05:18:25.129 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-16T05:18:25.129 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-16T05:18:25.129 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-16T05:18:25.129 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-16T05:18:25.129 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-karma.ko 2024-04-16T05:18:25.129 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-16T05:18:25.129 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-16T05:18:25.129 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-16T05:18:25.129 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-16T05:18:25.129 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-16T05:18:25.129 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/usb-storage.ko 2024-04-16T05:18:25.129 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip 2024-04-16T05:18:25.130 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-16T05:18:25.130 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-16T05:18:25.130 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-16T05:18:25.130 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio 2024-04-16T05:18:25.130 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci 2024-04-16T05:18:25.130 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-16T05:18:25.130 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-16T05:18:25.130 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio.ko 2024-04-16T05:18:25.130 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-16T05:18:25.130 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost 2024-04-16T05:18:25.130 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost.ko 2024-04-16T05:18:25.130 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-16T05:18:25.130 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_net.ko 2024-04-16T05:18:25.130 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_scsi.ko 2024-04-16T05:18:25.130 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video 2024-04-16T05:18:25.131 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight 2024-04-16T05:18:25.131 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/apple_bl.ko 2024-04-16T05:18:25.131 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lcd.ko 2024-04-16T05:18:25.131 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-16T05:18:25.131 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio 2024-04-16T05:18:25.131 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio.ko 2024-04-16T05:18:25.131 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_balloon.ko 2024-04-16T05:18:25.131 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_mmio.ko 2024-04-16T05:18:25.131 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci.ko 2024-04-16T05:18:25.131 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-16T05:18:25.131 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-16T05:18:25.131 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_ring.ko 2024-04-16T05:18:25.131 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1 2024-04-16T05:18:25.131 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters 2024-04-16T05:18:25.132 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2482.ko 2024-04-16T05:18:25.132 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2490.ko 2024-04-16T05:18:25.132 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves 2024-04-16T05:18:25.132 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-16T05:18:25.132 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-16T05:18:25.132 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-16T05:18:25.132 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-16T05:18:25.132 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-16T05:18:25.132 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-16T05:18:25.133 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-16T05:18:25.133 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-16T05:18:25.133 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-16T05:18:25.133 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-16T05:18:25.133 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-16T05:18:25.133 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/wire.ko 2024-04-16T05:18:25.133 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog 2024-04-16T05:18:25.133 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-16T05:18:25.133 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-16T05:18:25.133 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-16T05:18:25.133 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/hpwdt.ko 2024-04-16T05:18:25.133 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/i6300esb.ko 2024-04-16T05:18:25.134 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-16T05:18:25.134 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-16T05:18:25.134 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ib700wdt.ko 2024-04-16T05:18:25.134 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ibmasr.ko 2024-04-16T05:18:25.134 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-16T05:18:25.134 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-16T05:18:25.134 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it87_wdt.ko 2024-04-16T05:18:25.134 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/machzwd.ko 2024-04-16T05:18:25.134 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/nv_tco.ko 2024-04-16T05:18:25.134 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-16T05:18:25.134 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-16T05:18:25.134 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-16T05:18:25.135 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-16T05:18:25.135 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/softdog.ko 2024-04-16T05:18:25.135 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-16T05:18:25.135 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/via_wdt.ko 2024-04-16T05:18:25.135 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-16T05:18:25.135 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-16T05:18:25.135 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-16T05:18:25.135 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wdt_pci.ko 2024-04-16T05:18:25.135 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/xen_wdt.ko 2024-04-16T05:18:25.135 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen 2024-04-16T05:18:25.135 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-16T05:18:25.136 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-evtchn.ko 2024-04-16T05:18:25.136 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntalloc.ko 2024-04-16T05:18:25.136 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntdev.ko 2024-04-16T05:18:25.136 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback 2024-04-16T05:18:25.136 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-16T05:18:25.136 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-privcmd.ko 2024-04-16T05:18:25.136 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs 2024-04-16T05:18:25.136 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-16T05:18:25.136 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs 2024-04-16T05:18:25.136 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p 2024-04-16T05:18:25.136 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p/9p.ko 2024-04-16T05:18:25.137 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs 2024-04-16T05:18:25.137 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs/affs.ko 2024-04-16T05:18:25.137 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs 2024-04-16T05:18:25.137 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs/befs.ko 2024-04-16T05:18:25.137 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/binfmt_misc.ko 2024-04-16T05:18:25.137 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs 2024-04-16T05:18:25.137 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs/btrfs.ko 2024-04-16T05:18:25.137 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles 2024-04-16T05:18:25.137 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles/cachefiles.ko 2024-04-16T05:18:25.137 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph 2024-04-16T05:18:25.137 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph/ceph.ko 2024-04-16T05:18:25.137 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda 2024-04-16T05:18:25.137 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda/coda.ko 2024-04-16T05:18:25.137 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs 2024-04-16T05:18:25.137 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs/cramfs.ko 2024-04-16T05:18:25.138 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm 2024-04-16T05:18:25.138 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm/dlm.ko 2024-04-16T05:18:25.138 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs 2024-04-16T05:18:25.138 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-16T05:18:25.138 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs 2024-04-16T05:18:25.138 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs/f2fs.ko 2024-04-16T05:18:25.138 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat 2024-04-16T05:18:25.138 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/fat.ko 2024-04-16T05:18:25.138 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/msdos.ko 2024-04-16T05:18:25.138 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/vfat.ko 2024-04-16T05:18:25.138 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse 2024-04-16T05:18:25.138 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/cuse.ko 2024-04-16T05:18:25.138 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/fuse.ko 2024-04-16T05:18:25.139 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2 2024-04-16T05:18:25.139 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2/gfs2.ko 2024-04-16T05:18:25.139 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs 2024-04-16T05:18:25.139 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs/hfs.ko 2024-04-16T05:18:25.139 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus 2024-04-16T05:18:25.139 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus/hfsplus.ko 2024-04-16T05:18:25.139 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs 2024-04-16T05:18:25.139 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs/isofs.ko 2024-04-16T05:18:25.139 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs 2024-04-16T05:18:25.139 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs/jfs.ko 2024-04-16T05:18:25.139 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd 2024-04-16T05:18:25.139 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd/lockd.ko 2024-04-16T05:18:25.139 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix 2024-04-16T05:18:25.139 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix/minix.ko 2024-04-16T05:18:25.140 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs 2024-04-16T05:18:25.140 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs/netfs.ko 2024-04-16T05:18:25.140 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs 2024-04-16T05:18:25.140 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout 2024-04-16T05:18:25.140 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-16T05:18:25.140 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout 2024-04-16T05:18:25.140 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-16T05:18:25.140 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout 2024-04-16T05:18:25.140 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-16T05:18:25.140 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfs.ko 2024-04-16T05:18:25.140 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv3.ko 2024-04-16T05:18:25.140 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv4.ko 2024-04-16T05:18:25.140 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common 2024-04-16T05:18:25.140 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/grace.ko 2024-04-16T05:18:25.140 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/nfs_acl.ko 2024-04-16T05:18:25.141 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd 2024-04-16T05:18:25.141 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd/nfsd.ko 2024-04-16T05:18:25.141 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2 2024-04-16T05:18:25.141 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2/nilfs2.ko 2024-04-16T05:18:25.141 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls 2024-04-16T05:18:25.141 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-celtic.ko 2024-04-16T05:18:25.141 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-centeuro.ko 2024-04-16T05:18:25.141 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-croatian.ko 2024-04-16T05:18:25.141 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-cyrillic.ko 2024-04-16T05:18:25.141 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-gaelic.ko 2024-04-16T05:18:25.141 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-greek.ko 2024-04-16T05:18:25.141 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-iceland.ko 2024-04-16T05:18:25.141 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-inuit.ko 2024-04-16T05:18:25.141 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-roman.ko 2024-04-16T05:18:25.142 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-romanian.ko 2024-04-16T05:18:25.142 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-turkish.ko 2024-04-16T05:18:25.142 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1250.ko 2024-04-16T05:18:25.142 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1251.ko 2024-04-16T05:18:25.142 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1255.ko 2024-04-16T05:18:25.142 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp737.ko 2024-04-16T05:18:25.142 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp775.ko 2024-04-16T05:18:25.142 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp850.ko 2024-04-16T05:18:25.142 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp852.ko 2024-04-16T05:18:25.142 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp855.ko 2024-04-16T05:18:25.142 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp857.ko 2024-04-16T05:18:25.142 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp860.ko 2024-04-16T05:18:25.142 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp861.ko 2024-04-16T05:18:25.142 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp862.ko 2024-04-16T05:18:25.143 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp863.ko 2024-04-16T05:18:25.143 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp864.ko 2024-04-16T05:18:25.143 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp865.ko 2024-04-16T05:18:25.143 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp866.ko 2024-04-16T05:18:25.143 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp869.ko 2024-04-16T05:18:25.143 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp874.ko 2024-04-16T05:18:25.143 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp932.ko 2024-04-16T05:18:25.143 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp936.ko 2024-04-16T05:18:25.143 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp949.ko 2024-04-16T05:18:25.143 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp950.ko 2024-04-16T05:18:25.143 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_euc-jp.ko 2024-04-16T05:18:25.143 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-1.ko 2024-04-16T05:18:25.143 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-13.ko 2024-04-16T05:18:25.143 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-14.ko 2024-04-16T05:18:25.144 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-15.ko 2024-04-16T05:18:25.144 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-2.ko 2024-04-16T05:18:25.144 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-3.ko 2024-04-16T05:18:25.144 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-4.ko 2024-04-16T05:18:25.144 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-5.ko 2024-04-16T05:18:25.144 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-6.ko 2024-04-16T05:18:25.144 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-7.ko 2024-04-16T05:18:25.144 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-9.ko 2024-04-16T05:18:25.144 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-r.ko 2024-04-16T05:18:25.144 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-ru.ko 2024-04-16T05:18:25.144 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-u.ko 2024-04-16T05:18:25.144 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-16T05:18:25.144 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_utf8.ko 2024-04-16T05:18:25.144 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2 2024-04-16T05:18:25.145 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster 2024-04-16T05:18:25.145 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-16T05:18:25.145 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm 2024-04-16T05:18:25.145 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-16T05:18:25.145 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs 2024-04-16T05:18:25.145 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-16T05:18:25.145 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2.ko 2024-04-16T05:18:25.145 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-16T05:18:25.145 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-16T05:18:25.145 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-16T05:18:25.145 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs 2024-04-16T05:18:25.145 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs/overlay.ko 2024-04-16T05:18:25.145 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore 2024-04-16T05:18:25.145 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore/ramoops.ko 2024-04-16T05:18:25.145 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs 2024-04-16T05:18:25.146 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs/reiserfs.ko 2024-04-16T05:18:25.146 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs 2024-04-16T05:18:25.146 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs/romfs.ko 2024-04-16T05:18:25.146 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb 2024-04-16T05:18:25.146 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client 2024-04-16T05:18:25.146 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client/cifs.ko 2024-04-16T05:18:25.146 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common 2024-04-16T05:18:25.146 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_arc4.ko 2024-04-16T05:18:25.146 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_md4.ko 2024-04-16T05:18:25.146 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs 2024-04-16T05:18:25.146 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs/squashfs.ko 2024-04-16T05:18:25.146 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv 2024-04-16T05:18:25.146 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv/sysv.ko 2024-04-16T05:18:25.146 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf 2024-04-16T05:18:25.147 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf/udf.ko 2024-04-16T05:18:25.147 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs 2024-04-16T05:18:25.147 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs/ufs.ko 2024-04-16T05:18:25.147 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs 2024-04-16T05:18:25.147 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs/xfs.ko 2024-04-16T05:18:25.147 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel 2024-04-16T05:18:25.147 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu 2024-04-16T05:18:25.147 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu/rcutorture.ko 2024-04-16T05:18:25.147 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/torture.ko 2024-04-16T05:18:25.147 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace 2024-04-16T05:18:25.147 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-16T05:18:25.147 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib 2024-04-16T05:18:25.147 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/asn1_encoder.ko 2024-04-16T05:18:25.148 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc-itu-t.ko 2024-04-16T05:18:25.148 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc8.ko 2024-04-16T05:18:25.148 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto 2024-04-16T05:18:25.148 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libarc4.ko 2024-04-16T05:18:25.148 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libdes.ko 2024-04-16T05:18:25.148 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/libcrc32c.ko 2024-04-16T05:18:25.148 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lru_cache.ko 2024-04-16T05:18:25.148 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4 2024-04-16T05:18:25.148 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4_compress.ko 2024-04-16T05:18:25.148 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4hc_compress.ko 2024-04-16T05:18:25.149 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math 2024-04-16T05:18:25.149 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math/cordic.ko 2024-04-16T05:18:25.149 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6 2024-04-16T05:18:25.149 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6/raid6_pq.ko 2024-04-16T05:18:25.149 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon 2024-04-16T05:18:25.149 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-16T05:18:25.149 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_bm.ko 2024-04-16T05:18:25.149 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_fsm.ko 2024-04-16T05:18:25.149 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_kmp.ko 2024-04-16T05:18:25.149 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd 2024-04-16T05:18:25.149 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd/zstd_compress.ko 2024-04-16T05:18:25.149 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm 2024-04-16T05:18:25.149 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm/hwpoison-inject.ko 2024-04-16T05:18:25.149 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net 2024-04-16T05:18:25.150 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan 2024-04-16T05:18:25.150 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/6lowpan.ko 2024-04-16T05:18:25.150 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_dest.ko 2024-04-16T05:18:25.150 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_fragment.ko 2024-04-16T05:18:25.150 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_hop.ko 2024-04-16T05:18:25.150 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-16T05:18:25.150 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_mobility.ko 2024-04-16T05:18:25.150 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_routing.ko 2024-04-16T05:18:25.150 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_udp.ko 2024-04-16T05:18:25.150 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802 2024-04-16T05:18:25.150 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/garp.ko 2024-04-16T05:18:25.150 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/mrp.ko 2024-04-16T05:18:25.150 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/p8022.ko 2024-04-16T05:18:25.150 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/psnap.ko 2024-04-16T05:18:25.151 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/stp.ko 2024-04-16T05:18:25.151 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q 2024-04-16T05:18:25.151 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q/8021q.ko 2024-04-16T05:18:25.151 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p 2024-04-16T05:18:25.151 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet.ko 2024-04-16T05:18:25.151 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_fd.ko 2024-04-16T05:18:25.151 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_rdma.ko 2024-04-16T05:18:25.151 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_virtio.ko 2024-04-16T05:18:25.151 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv 2024-04-16T05:18:25.151 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv/batman-adv.ko 2024-04-16T05:18:25.151 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge 2024-04-16T05:18:25.151 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/br_netfilter.ko 2024-04-16T05:18:25.151 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/bridge.ko 2024-04-16T05:18:25.151 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter 2024-04-16T05:18:25.152 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-16T05:18:25.152 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-16T05:18:25.152 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-16T05:18:25.152 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-16T05:18:25.152 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-16T05:18:25.152 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-16T05:18:25.152 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-16T05:18:25.152 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-16T05:18:25.152 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-16T05:18:25.152 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-16T05:18:25.152 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-16T05:18:25.152 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-16T05:18:25.152 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-16T05:18:25.152 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-16T05:18:25.153 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-16T05:18:25.153 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-16T05:18:25.153 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-16T05:18:25.153 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-16T05:18:25.153 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-16T05:18:25.153 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-16T05:18:25.153 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtables.ko 2024-04-16T05:18:25.153 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-16T05:18:25.153 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-16T05:18:25.153 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-16T05:18:25.153 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph 2024-04-16T05:18:25.153 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph/libceph.ko 2024-04-16T05:18:25.153 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core 2024-04-16T05:18:25.153 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/failover.ko 2024-04-16T05:18:25.153 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/pktgen.ko 2024-04-16T05:18:25.154 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp 2024-04-16T05:18:25.154 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp.ko 2024-04-16T05:18:25.154 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_diag.ko 2024-04-16T05:18:25.154 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv4.ko 2024-04-16T05:18:25.154 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv6.ko 2024-04-16T05:18:25.154 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver 2024-04-16T05:18:25.154 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver/dns_resolver.ko 2024-04-16T05:18:25.154 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa 2024-04-16T05:18:25.154 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/dsa_core.ko 2024-04-16T05:18:25.154 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_brcm.ko 2024-04-16T05:18:25.154 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_dsa.ko 2024-04-16T05:18:25.154 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_none.ko 2024-04-16T05:18:25.154 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_trailer.ko 2024-04-16T05:18:25.154 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154 2024-04-16T05:18:25.155 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan 2024-04-16T05:18:25.155 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-16T05:18:25.155 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154.ko 2024-04-16T05:18:25.155 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-16T05:18:25.155 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4 2024-04-16T05:18:25.155 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ah4.ko 2024-04-16T05:18:25.155 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/esp4.ko 2024-04-16T05:18:25.155 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/fou.ko 2024-04-16T05:18:25.155 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/gre.ko 2024-04-16T05:18:25.155 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/inet_diag.ko 2024-04-16T05:18:25.155 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_gre.ko 2024-04-16T05:18:25.155 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_tunnel.ko 2024-04-16T05:18:25.155 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_vti.ko 2024-04-16T05:18:25.155 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipcomp.ko 2024-04-16T05:18:25.156 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipip.ko 2024-04-16T05:18:25.156 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter 2024-04-16T05:18:25.156 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-16T05:18:25.156 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-16T05:18:25.156 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-16T05:18:25.156 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-16T05:18:25.156 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-16T05:18:25.156 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-16T05:18:25.156 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-16T05:18:25.156 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-16T05:18:25.156 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-16T05:18:25.156 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-16T05:18:25.156 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-16T05:18:25.156 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-16T05:18:25.157 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-16T05:18:25.157 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-16T05:18:25.157 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-16T05:18:25.157 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-16T05:18:25.157 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-16T05:18:25.157 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-16T05:18:25.157 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-16T05:18:25.157 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-16T05:18:25.157 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-16T05:18:25.157 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_bic.ko 2024-04-16T05:18:25.157 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_dctcp.ko 2024-04-16T05:18:25.157 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_diag.ko 2024-04-16T05:18:25.157 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_highspeed.ko 2024-04-16T05:18:25.157 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_htcp.ko 2024-04-16T05:18:25.157 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_hybla.ko 2024-04-16T05:18:25.158 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_illinois.ko 2024-04-16T05:18:25.158 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_lp.ko 2024-04-16T05:18:25.158 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_scalable.ko 2024-04-16T05:18:25.158 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_vegas.ko 2024-04-16T05:18:25.158 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_veno.ko 2024-04-16T05:18:25.158 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_westwood.ko 2024-04-16T05:18:25.158 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_yeah.ko 2024-04-16T05:18:25.158 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tunnel4.ko 2024-04-16T05:18:25.158 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_diag.ko 2024-04-16T05:18:25.158 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_tunnel.ko 2024-04-16T05:18:25.158 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-16T05:18:25.158 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6 2024-04-16T05:18:25.158 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ah6.ko 2024-04-16T05:18:25.159 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/esp6.ko 2024-04-16T05:18:25.159 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/fou6.ko 2024-04-16T05:18:25.159 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_tunnel.ko 2024-04-16T05:18:25.159 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-16T05:18:25.159 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_vti.ko 2024-04-16T05:18:25.159 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ipcomp6.ko 2024-04-16T05:18:25.159 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter 2024-04-16T05:18:25.159 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-16T05:18:25.159 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-16T05:18:25.159 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-16T05:18:25.159 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-16T05:18:25.159 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-16T05:18:25.159 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-16T05:18:25.159 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-16T05:18:25.159 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-16T05:18:25.160 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-16T05:18:25.160 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-16T05:18:25.160 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-16T05:18:25.160 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-16T05:18:25.160 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-16T05:18:25.160 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-16T05:18:25.160 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-16T05:18:25.160 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-16T05:18:25.160 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-16T05:18:25.160 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-16T05:18:25.160 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-16T05:18:25.160 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-16T05:18:25.160 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-16T05:18:25.160 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-16T05:18:25.160 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-16T05:18:25.161 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-16T05:18:25.161 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/sit.ko 2024-04-16T05:18:25.161 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/tunnel6.ko 2024-04-16T05:18:25.161 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-16T05:18:25.161 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key 2024-04-16T05:18:25.161 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key/af_key.ko 2024-04-16T05:18:25.161 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp 2024-04-16T05:18:25.161 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_core.ko 2024-04-16T05:18:25.161 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-16T05:18:25.161 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_eth.ko 2024-04-16T05:18:25.161 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip.ko 2024-04-16T05:18:25.161 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip6.ko 2024-04-16T05:18:25.161 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_netlink.ko 2024-04-16T05:18:25.161 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ppp.ko 2024-04-16T05:18:25.162 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc 2024-04-16T05:18:25.162 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/llc.ko 2024-04-16T05:18:25.162 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154 2024-04-16T05:18:25.162 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154/mac802154.ko 2024-04-16T05:18:25.162 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls 2024-04-16T05:18:25.162 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls/mpls_gso.ko 2024-04-16T05:18:25.162 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter 2024-04-16T05:18:25.162 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset 2024-04-16T05:18:25.162 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set.ko 2024-04-16T05:18:25.162 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-16T05:18:25.162 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-16T05:18:25.162 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-16T05:18:25.162 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-16T05:18:25.162 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-16T05:18:25.163 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-16T05:18:25.163 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-16T05:18:25.163 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-16T05:18:25.163 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-16T05:18:25.163 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-16T05:18:25.163 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-16T05:18:25.163 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-16T05:18:25.163 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-16T05:18:25.163 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-16T05:18:25.163 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-16T05:18:25.163 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs 2024-04-16T05:18:25.163 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-16T05:18:25.163 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-16T05:18:25.163 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-16T05:18:25.163 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-16T05:18:25.164 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-16T05:18:25.164 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-16T05:18:25.164 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-16T05:18:25.164 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-16T05:18:25.164 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-16T05:18:25.164 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-16T05:18:25.164 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-16T05:18:25.164 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-16T05:18:25.164 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-16T05:18:25.164 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-16T05:18:25.164 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conncount.ko 2024-04-16T05:18:25.164 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack.ko 2024-04-16T05:18:25.164 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-16T05:18:25.164 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-16T05:18:25.165 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-16T05:18:25.165 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-16T05:18:25.165 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-16T05:18:25.165 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-16T05:18:25.165 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-16T05:18:25.165 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-16T05:18:25.165 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-16T05:18:25.165 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-16T05:18:25.165 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-16T05:18:25.165 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-16T05:18:25.165 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-16T05:18:25.165 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table.ko 2024-04-16T05:18:25.165 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-16T05:18:25.165 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_log_syslog.ko 2024-04-16T05:18:25.165 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat.ko 2024-04-16T05:18:25.166 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-16T05:18:25.166 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-16T05:18:25.166 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_irc.ko 2024-04-16T05:18:25.166 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_sip.ko 2024-04-16T05:18:25.166 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-16T05:18:25.166 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-16T05:18:25.166 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_tables.ko 2024-04-16T05:18:25.166 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink.ko 2024-04-16T05:18:25.166 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-16T05:18:25.166 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_log.ko 2024-04-16T05:18:25.166 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-16T05:18:25.166 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-16T05:18:25.166 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_chain_nat.ko 2024-04-16T05:18:25.166 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_compat.ko 2024-04-16T05:18:25.167 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_ct.ko 2024-04-16T05:18:25.167 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-16T05:18:25.167 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fib.ko 2024-04-16T05:18:25.167 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-16T05:18:25.167 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_hash.ko 2024-04-16T05:18:25.167 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_limit.ko 2024-04-16T05:18:25.167 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_log.ko 2024-04-16T05:18:25.167 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_masq.ko 2024-04-16T05:18:25.167 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_nat.ko 2024-04-16T05:18:25.167 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_numgen.ko 2024-04-16T05:18:25.167 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_queue.ko 2024-04-16T05:18:25.167 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_quota.ko 2024-04-16T05:18:25.167 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_redir.ko 2024-04-16T05:18:25.167 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject.ko 2024-04-16T05:18:25.168 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject_inet.ko 2024-04-16T05:18:25.168 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_socket.ko 2024-04-16T05:18:25.168 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_tunnel.ko 2024-04-16T05:18:25.168 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_AUDIT.ko 2024-04-16T05:18:25.168 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-16T05:18:25.168 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-16T05:18:25.168 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-16T05:18:25.168 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CT.ko 2024-04-16T05:18:25.168 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_DSCP.ko 2024-04-16T05:18:25.168 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HL.ko 2024-04-16T05:18:25.168 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HMARK.ko 2024-04-16T05:18:25.168 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-16T05:18:25.168 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LED.ko 2024-04-16T05:18:25.168 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LOG.ko 2024-04-16T05:18:25.168 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-16T05:18:25.169 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NETMAP.ko 2024-04-16T05:18:25.169 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFLOG.ko 2024-04-16T05:18:25.169 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-16T05:18:25.169 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_RATEEST.ko 2024-04-16T05:18:25.169 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-16T05:18:25.169 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_SECMARK.ko 2024-04-16T05:18:25.169 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-16T05:18:25.169 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-16T05:18:25.169 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TEE.ko 2024-04-16T05:18:25.169 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TPROXY.ko 2024-04-16T05:18:25.169 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TRACE.ko 2024-04-16T05:18:25.169 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_addrtype.ko 2024-04-16T05:18:25.169 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_bpf.ko 2024-04-16T05:18:25.169 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cgroup.ko 2024-04-16T05:18:25.169 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cluster.ko 2024-04-16T05:18:25.170 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_comment.ko 2024-04-16T05:18:25.170 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connbytes.ko 2024-04-16T05:18:25.170 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlabel.ko 2024-04-16T05:18:25.170 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlimit.ko 2024-04-16T05:18:25.170 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connmark.ko 2024-04-16T05:18:25.170 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_conntrack.ko 2024-04-16T05:18:25.170 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cpu.ko 2024-04-16T05:18:25.170 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dccp.ko 2024-04-16T05:18:25.170 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_devgroup.ko 2024-04-16T05:18:25.170 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dscp.ko 2024-04-16T05:18:25.170 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ecn.ko 2024-04-16T05:18:25.170 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_esp.ko 2024-04-16T05:18:25.170 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hashlimit.ko 2024-04-16T05:18:25.170 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_helper.ko 2024-04-16T05:18:25.171 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hl.ko 2024-04-16T05:18:25.171 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipcomp.ko 2024-04-16T05:18:25.171 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_iprange.ko 2024-04-16T05:18:25.171 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipvs.ko 2024-04-16T05:18:25.171 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_l2tp.ko 2024-04-16T05:18:25.171 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_length.ko 2024-04-16T05:18:25.171 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_limit.ko 2024-04-16T05:18:25.173 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mac.ko 2024-04-16T05:18:25.173 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mark.ko 2024-04-16T05:18:25.173 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_multiport.ko 2024-04-16T05:18:25.174 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nat.ko 2024-04-16T05:18:25.174 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nfacct.ko 2024-04-16T05:18:25.174 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_osf.ko 2024-04-16T05:18:25.174 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_owner.ko 2024-04-16T05:18:25.174 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_physdev.ko 2024-04-16T05:18:25.174 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_pkttype.ko 2024-04-16T05:18:25.174 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_policy.ko 2024-04-16T05:18:25.174 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_quota.ko 2024-04-16T05:18:25.174 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_rateest.ko 2024-04-16T05:18:25.174 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_realm.ko 2024-04-16T05:18:25.174 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_recent.ko 2024-04-16T05:18:25.174 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_sctp.ko 2024-04-16T05:18:25.174 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_set.ko 2024-04-16T05:18:25.174 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_socket.ko 2024-04-16T05:18:25.175 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_state.ko 2024-04-16T05:18:25.175 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_statistic.ko 2024-04-16T05:18:25.175 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_string.ko 2024-04-16T05:18:25.175 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_tcpmss.ko 2024-04-16T05:18:25.175 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_time.ko 2024-04-16T05:18:25.175 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_u32.ko 2024-04-16T05:18:25.175 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink 2024-04-16T05:18:25.175 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink/netlink_diag.ko 2024-04-16T05:18:25.175 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh 2024-04-16T05:18:25.175 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh/nsh.ko 2024-04-16T05:18:25.175 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch 2024-04-16T05:18:25.175 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/openvswitch.ko 2024-04-16T05:18:25.175 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-geneve.ko 2024-04-16T05:18:25.175 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-gre.ko 2024-04-16T05:18:25.176 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-vxlan.ko 2024-04-16T05:18:25.176 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet 2024-04-16T05:18:25.176 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet/af_packet_diag.ko 2024-04-16T05:18:25.176 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds 2024-04-16T05:18:25.176 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds.ko 2024-04-16T05:18:25.176 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_rdma.ko 2024-04-16T05:18:25.176 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_tcp.ko 2024-04-16T05:18:25.176 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched 2024-04-16T05:18:25.176 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_csum.ko 2024-04-16T05:18:25.176 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_gact.ko 2024-04-16T05:18:25.176 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_mirred.ko 2024-04-16T05:18:25.176 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_nat.ko 2024-04-16T05:18:25.176 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_pedit.ko 2024-04-16T05:18:25.176 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_police.ko 2024-04-16T05:18:25.176 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_simple.ko 2024-04-16T05:18:25.177 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_skbedit.ko 2024-04-16T05:18:25.177 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_vlan.ko 2024-04-16T05:18:25.177 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_basic.ko 2024-04-16T05:18:25.177 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_bpf.ko 2024-04-16T05:18:25.177 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_flow.ko 2024-04-16T05:18:25.177 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_fw.ko 2024-04-16T05:18:25.177 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_route.ko 2024-04-16T05:18:25.177 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_u32.ko 2024-04-16T05:18:25.177 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_cmp.ko 2024-04-16T05:18:25.177 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_ipset.ko 2024-04-16T05:18:25.177 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_meta.ko 2024-04-16T05:18:25.177 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_nbyte.ko 2024-04-16T05:18:25.177 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_text.ko 2024-04-16T05:18:25.177 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_u32.ko 2024-04-16T05:18:25.178 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_choke.ko 2024-04-16T05:18:25.178 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_codel.ko 2024-04-16T05:18:25.178 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_drr.ko 2024-04-16T05:18:25.178 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_fq.ko 2024-04-16T05:18:25.178 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_gred.ko 2024-04-16T05:18:25.178 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hfsc.ko 2024-04-16T05:18:25.178 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hhf.ko 2024-04-16T05:18:25.178 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_htb.ko 2024-04-16T05:18:25.178 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_ingress.ko 2024-04-16T05:18:25.178 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio.ko 2024-04-16T05:18:25.178 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio_lib.ko 2024-04-16T05:18:25.178 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_multiq.ko 2024-04-16T05:18:25.178 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_netem.ko 2024-04-16T05:18:25.178 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_pie.ko 2024-04-16T05:18:25.179 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_plug.ko 2024-04-16T05:18:25.179 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_prio.ko 2024-04-16T05:18:25.179 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_qfq.ko 2024-04-16T05:18:25.179 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_red.ko 2024-04-16T05:18:25.179 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfb.ko 2024-04-16T05:18:25.179 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfq.ko 2024-04-16T05:18:25.179 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_tbf.ko 2024-04-16T05:18:25.179 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_teql.ko 2024-04-16T05:18:25.179 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp 2024-04-16T05:18:25.179 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp.ko 2024-04-16T05:18:25.179 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp_diag.ko 2024-04-16T05:18:25.179 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc 2024-04-16T05:18:25.179 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss 2024-04-16T05:18:25.179 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-16T05:18:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-16T05:18:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/sunrpc.ko 2024-04-16T05:18:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma 2024-04-16T05:18:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-16T05:18:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc 2024-04-16T05:18:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc.ko 2024-04-16T05:18:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc_diag.ko 2024-04-16T05:18:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix 2024-04-16T05:18:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix/unix_diag.ko 2024-04-16T05:18:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock 2024-04-16T05:18:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-16T05:18:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-16T05:18:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock.ko 2024-04-16T05:18:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-16T05:18:25.180 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-16T05:18:25.181 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm 2024-04-16T05:18:25.181 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-16T05:18:25.181 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security 2024-04-16T05:18:25.181 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys 2024-04-16T05:18:25.181 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys 2024-04-16T05:18:25.181 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-16T05:18:25.181 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys 2024-04-16T05:18:25.181 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys/trusted.ko 2024-04-16T05:18:25.181 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias 2024-04-16T05:18:25.181 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias.bin 2024-04-16T05:18:25.181 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin 2024-04-16T05:18:25.181 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.alias.bin 2024-04-16T05:18:25.181 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.bin 2024-04-16T05:18:25.181 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.modinfo 2024-04-16T05:18:25.182 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep 2024-04-16T05:18:25.182 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep.bin 2024-04-16T05:18:25.182 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.devname 2024-04-16T05:18:25.182 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.order 2024-04-16T05:18:25.182 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.softdep 2024-04-16T05:18:25.182 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols 2024-04-16T05:18:25.182 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols.bin 2024-04-16T05:18:25.182 INFO:teuthology.orchestra.run.smithi122.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-16T05:18:25.186 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T05:18:25.186 DEBUG:teuthology.orchestra.run.smithi122:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-16T05:18:25.209 INFO:teuthology.orchestra.run.smithi122.stdout:postinstall scriptlet (using /bin/sh): 2024-04-16T05:18:25.209 INFO:teuthology.orchestra.run.smithi122.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-16T05:18:25.209 INFO:teuthology.orchestra.run.smithi122.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g3f60ed14f7c1 /lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-16T05:18:25.209 INFO:teuthology.orchestra.run.smithi122.stdout:fi 2024-04-16T05:18:25.209 INFO:teuthology.orchestra.run.smithi122.stdout:for file in vmlinuz System.map config; do 2024-04-16T05:18:25.209 INFO:teuthology.orchestra.run.smithi122.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1"; then 2024-04-16T05:18:25.209 INFO:teuthology.orchestra.run.smithi122.stdout: cp "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T05:18:25.209 INFO:teuthology.orchestra.run.smithi122.stdout: fi 2024-04-16T05:18:25.210 INFO:teuthology.orchestra.run.smithi122.stdout:done 2024-04-16T05:18:25.210 INFO:teuthology.orchestra.run.smithi122.stdout:if [ -d "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" ] && \ 2024-04-16T05:18:25.210 INFO:teuthology.orchestra.run.smithi122.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" >/dev/null 2>&1; then 2024-04-16T05:18:25.210 INFO:teuthology.orchestra.run.smithi122.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T05:18:25.210 INFO:teuthology.orchestra.run.smithi122.stdout: cp -r "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-16T05:18:25.210 INFO:teuthology.orchestra.run.smithi122.stdout:fi 2024-04-16T05:18:25.210 INFO:teuthology.orchestra.run.smithi122.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep" ]; then 2024-04-16T05:18:25.210 INFO:teuthology.orchestra.run.smithi122.stdout: /usr/sbin/depmod 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T05:18:25.210 INFO:teuthology.orchestra.run.smithi122.stdout:fi 2024-04-16T05:18:25.210 INFO:teuthology.orchestra.run.smithi122.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-16T05:18:25.210 INFO:teuthology.orchestra.run.smithi122.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-16T05:18:25.210 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-install remove 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T05:18:25.210 INFO:teuthology.orchestra.run.smithi122.stdout:fi 2024-04-16T05:18:25.211 DEBUG:teuthology.orchestra.run.smithi122:> sudo rpm -qi grub2-tools 2024-04-16T05:18:25.283 INFO:teuthology.orchestra.run.smithi122.stdout:Name : grub2-tools 2024-04-16T05:18:25.283 INFO:teuthology.orchestra.run.smithi122.stdout:Epoch : 1 2024-04-16T05:18:25.283 INFO:teuthology.orchestra.run.smithi122.stdout:Version : 2.06 2024-04-16T05:18:25.283 INFO:teuthology.orchestra.run.smithi122.stdout:Release : 68.el9 2024-04-16T05:18:25.283 INFO:teuthology.orchestra.run.smithi122.stdout:Architecture: x86_64 2024-04-16T05:18:25.283 INFO:teuthology.orchestra.run.smithi122.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T05:18:25.283 INFO:teuthology.orchestra.run.smithi122.stdout:Group : Unspecified 2024-04-16T05:18:25.283 INFO:teuthology.orchestra.run.smithi122.stdout:Size : 8264254 2024-04-16T05:18:25.283 INFO:teuthology.orchestra.run.smithi122.stdout:License : GPLv3+ 2024-04-16T05:18:25.283 INFO:teuthology.orchestra.run.smithi122.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T05:18:25.284 INFO:teuthology.orchestra.run.smithi122.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T05:18:25.284 INFO:teuthology.orchestra.run.smithi122.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T05:18:25.284 INFO:teuthology.orchestra.run.smithi122.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T05:18:25.284 INFO:teuthology.orchestra.run.smithi122.stdout:Packager : builder@centos.org 2024-04-16T05:18:25.284 INFO:teuthology.orchestra.run.smithi122.stdout:Vendor : CentOS 2024-04-16T05:18:25.284 INFO:teuthology.orchestra.run.smithi122.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T05:18:25.284 INFO:teuthology.orchestra.run.smithi122.stdout:Summary : Support tools for GRUB. 2024-04-16T05:18:25.284 INFO:teuthology.orchestra.run.smithi122.stdout:Description : 2024-04-16T05:18:25.284 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:18:25.284 INFO:teuthology.orchestra.run.smithi122.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T05:18:25.284 INFO:teuthology.orchestra.run.smithi122.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T05:18:25.284 INFO:teuthology.orchestra.run.smithi122.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T05:18:25.284 INFO:teuthology.orchestra.run.smithi122.stdout:hardware devices. 2024-04-16T05:18:25.284 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:18:25.285 INFO:teuthology.orchestra.run.smithi122.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T05:18:25.285 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T05:18:25.285 INFO:teuthology.task.kernel:Updating grub on smithi122 to boot 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T05:18:25.286 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T05:18:25.842 INFO:teuthology.orchestra.run.smithi122.stderr:Generating grub configuration file ... 2024-04-16T05:18:27.236 INFO:teuthology.orchestra.run.smithi122.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T05:18:27.272 INFO:teuthology.orchestra.run.smithi122.stderr:done 2024-04-16T05:18:27.275 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-04-16T05:18:27.290 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.jk9i3cq5At 2024-04-16T05:18:27.291 DEBUG:teuthology.orchestra.run.smithi122:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.jk9i3cq5At 2024-04-16T05:18:27.360 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.jk9i3cq5At 2024-04-16T05:18:27.505 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.jk9i3cq5At is 6KB 2024-04-16T05:18:27.554 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.jk9i3cq5At 2024-04-16T05:18:27.568 DEBUG:teuthology.orchestra.run.smithi122:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T05:18:27.664 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T05:18:27.665 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T05:18:27.665 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T05:18:27.665 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1.conf 2024-04-16T05:18:27.666 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1 2024-04-16T05:18:27.797 DEBUG:teuthology.orchestra.run.smithi122:> sudo shutdown -r now 2024-04-16T05:18:29.500 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-04-16T05:18:29.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-04-16T05:18:29.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:18:57.828 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-04-16T05:18:57.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-16T05:18:57.829 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:19:03.966 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-04-16T05:19:12.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-04-16T05:19:12.977 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:19:16.030 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-04-16T05:19:16.254 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-04-16T05:19:25.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-16T05:19:25.265 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:19:28.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-04-16T05:19:28.034 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:19:28.322 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.122 2024-04-16T05:19:31.102 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-04-16T05:19:40.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-16T05:19:40.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:19:46.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-04-16T05:19:46.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:20:04.638 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-04-16T05:20:12.148 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-04-16T05:20:12.742 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-16T05:20:12.742 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "6.9.0-rc3-g3f60ed14f7c1"... 2024-04-16T05:20:12.743 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-04-16T05:20:12.781 INFO:teuthology.orchestra.run.smithi122.stdout:6.9.0-rc3-g3f60ed14f7c1 2024-04-16T05:20:12.781 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g3f60ed14f7c1 vs 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T05:20:12.782 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T05:20:12.782 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T05:20:13.783 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-16T05:20:13.783 DEBUG:teuthology.orchestra.run.smithi122:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T05:20:13.923 INFO:teuthology.orchestra.run.smithi122.stdout:ttyS1 2024-04-16T05:20:13.965 DEBUG:teuthology.parallel:result is None 2024-04-16T05:20:22.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-04-16T05:20:22.640 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:20:23.184 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-04-16T05:20:23.879 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-04-16T05:20:23.879 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "6.9.0-rc3-g3f60ed14f7c1"... 2024-04-16T05:20:23.879 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-04-16T05:20:23.932 INFO:teuthology.orchestra.run.smithi027.stdout:6.9.0-rc3-g3f60ed14f7c1 2024-04-16T05:20:23.933 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g3f60ed14f7c1 vs 6.9.0-rc3-g3f60ed14f7c1 2024-04-16T05:20:23.933 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T05:20:23.933 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T05:20:24.933 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-16T05:20:24.934 DEBUG:teuthology.orchestra.run.smithi027:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T05:20:25.090 INFO:teuthology.orchestra.run.smithi027.stdout:ttyS1 2024-04-16T05:20:25.137 DEBUG:teuthology.parallel:result is None 2024-04-16T05:20:25.138 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-16T05:20:25.151 INFO:teuthology.task.internal:Creating test directory... 2024-04-16T05:20:25.151 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T05:20:25.158 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T05:20:25.242 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-16T05:20:25.250 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-16T05:20:25.258 INFO:teuthology.task.internal:Creating archive directory... 2024-04-16T05:20:25.258 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T05:20:25.263 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T05:20:25.365 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-16T05:20:25.372 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-16T05:20:25.372 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T05:20:25.378 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T05:20:25.489 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T05:20:25.519 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T05:20:25.541 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T05:20:25.576 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T05:20:25.584 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-16T05:20:25.591 INFO:teuthology.task.internal:Configuring sudo... 2024-04-16T05:20:25.592 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T05:20:25.597 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T05:20:25.736 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-16T05:20:25.745 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-16T05:20:25.745 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T05:20:25.749 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T05:20:25.831 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T05:20:25.911 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T05:20:26.025 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T05:20:26.101 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T05:20:26.181 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-16T05:20:26.181 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T05:20:26.258 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T05:20:26.371 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T05:20:26.499 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T05:20:26.565 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T05:20:26.651 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-16T05:20:26.651 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T05:20:26.720 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2024-04-16T05:20:26.727 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2024-04-16T05:20:26.846 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T05:20:26.872 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T05:20:27.370 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-16T05:20:27.378 INFO:teuthology.task.internal:Starting timer... 2024-04-16T05:20:27.378 INFO:teuthology.run_tasks:Running task pcp... 2024-04-16T05:20:27.394 INFO:teuthology.run_tasks:Running task selinux... 2024-04-16T05:20:27.404 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-16T05:20:27.405 DEBUG:teuthology.orchestra.run.smithi027:> sudo service auditd rotate 2024-04-16T05:20:27.515 INFO:teuthology.orchestra.run.smithi027.stdout:Rotating logs: 2024-04-16T05:20:27.520 DEBUG:teuthology.orchestra.run.smithi122:> sudo service auditd rotate 2024-04-16T05:20:27.649 INFO:teuthology.orchestra.run.smithi122.stdout:Rotating logs: 2024-04-16T05:20:27.653 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-16T05:20:27.653 DEBUG:teuthology.orchestra.run.smithi027:> /usr/sbin/getenforce 2024-04-16T05:20:27.703 INFO:teuthology.orchestra.run.smithi027.stdout:Permissive 2024-04-16T05:20:27.704 DEBUG:teuthology.orchestra.run.smithi122:> /usr/sbin/getenforce 2024-04-16T05:20:27.748 INFO:teuthology.orchestra.run.smithi122.stdout:Permissive 2024-04-16T05:20:27.749 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi027.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi122.front.sepia.ceph.com': 'permissive'} 2024-04-16T05:20:27.749 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T05:20:27.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T05:20:27.825 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T05:20:27.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T05:20:27.904 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-16T05:20:27.904 DEBUG:teuthology.orchestra.run.smithi027:> sudo /usr/sbin/setenforce permissive 2024-04-16T05:20:27.993 DEBUG:teuthology.orchestra.run.smithi122:> sudo /usr/sbin/setenforce permissive 2024-04-16T05:20:28.078 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-16T05:20:28.087 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-16T05:20:28.093 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-16T05:20:28.234 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-16T05:20:28.250 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-16T05:20:28.254 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi027.front.sepia.ceph.com,smithi122.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-16T05:26:45.573 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi027.front.sepia.ceph.com'), Remote(name='ubuntu@smithi122.front.sepia.ceph.com')] 2024-04-16T05:26:45.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-04-16T05:26:45.575 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:26:45.735 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-04-16T05:26:45.897 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-04-16T05:26:45.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-16T05:26:45.898 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:26:46.033 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-04-16T05:26:46.190 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-04-16T05:26:46.190 INFO:teuthology.run_tasks:Running task clock... 2024-04-16T05:26:46.202 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-16T05:26:46.203 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T05:26:46.203 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T05:26:46.211 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T05:26:46.211 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T05:26:46.313 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T05:26:46.321 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T05:26:46.365 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T05:26:46.383 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T05:26:46.464 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: ntpd: command not found 2024-04-16T05:26:46.491 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: ntpd: command not found 2024-04-16T05:26:46.507 INFO:teuthology.orchestra.run.smithi122.stdout:506 Cannot talk to daemon 2024-04-16T05:26:46.544 INFO:teuthology.orchestra.run.smithi027.stdout:506 Cannot talk to daemon 2024-04-16T05:26:46.559 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T05:26:46.606 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T05:26:46.610 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T05:26:46.666 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T05:26:46.753 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-04-16T05:26:46.760 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T05:26:46.760 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-04-16T05:26:46.760 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T05:26:46.760 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T05:26:46.761 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T05:26:46.761 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T05:26:46.817 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-04-16T05:26:46.828 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T05:26:46.828 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-04-16T05:26:46.828 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T05:26:46.828 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T05:26:46.828 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T05:26:46.829 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T05:26:46.831 INFO:teuthology.run_tasks:Running task install... 2024-04-16T05:26:46.840 DEBUG:teuthology.task.install:project ceph 2024-04-16T05:26:46.840 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838'}} 2024-04-16T05:26:46.840 DEBUG:teuthology.task.install:config {'branch': 'wip-64502-quincy-3', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838'} 2024-04-16T05:26:46.840 INFO:teuthology.task.install:Using flavor: default 2024-04-16T05:26:46.846 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-16T05:26:46.846 INFO:teuthology.task.install:extra packages: [] 2024-04-16T05:26:46.846 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'wip-64502-quincy-3', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838', 'tag': None, 'wait_for_package': False} 2024-04-16T05:26:46.847 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T05:26:46.847 INFO:teuthology.packaging:ref: None 2024-04-16T05:26:46.847 INFO:teuthology.packaging:tag: None 2024-04-16T05:26:46.847 INFO:teuthology.packaging:branch: wip-64502-quincy-3 2024-04-16T05:26:46.847 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T05:26:46.847 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-64502-quincy-3 2024-04-16T05:26:46.850 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'wip-64502-quincy-3', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838', 'tag': None, 'wait_for_package': False} 2024-04-16T05:26:46.850 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T05:26:46.850 INFO:teuthology.packaging:ref: None 2024-04-16T05:26:46.850 INFO:teuthology.packaging:tag: None 2024-04-16T05:26:46.850 INFO:teuthology.packaging:branch: wip-64502-quincy-3 2024-04-16T05:26:46.850 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T05:26:46.850 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-64502-quincy-3 2024-04-16T05:26:46.997 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/ 2024-04-16T05:26:46.997 INFO:teuthology.task.install.rpm:Package version is 17.2.7-847.g45317f95 2024-04-16T05:26:47.076 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/ 2024-04-16T05:26:47.076 INFO:teuthology.task.install.rpm:Package version is 17.2.7-847.g45317f95 2024-04-16T05:26:47.174 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T05:26:47.174 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-16T05:26:47.174 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T05:26:47.252 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-64502-quincy-3/45317f95c5587af98fd8eaefff9013a4ba51d2f7/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T05:26:47.252 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-16T05:26:47.252 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T05:26:47.288 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T05:26:47.288 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T05:26:47.288 INFO:teuthology.packaging:ref: None 2024-04-16T05:26:47.288 INFO:teuthology.packaging:tag: None 2024-04-16T05:26:47.288 INFO:teuthology.packaging:branch: wip-64502-quincy-3 2024-04-16T05:26:47.288 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T05:26:47.288 DEBUG:teuthology.orchestra.run.smithi027:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/wip-64502-quincy-3/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T05:26:47.352 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T05:26:47.352 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T05:26:47.352 INFO:teuthology.packaging:ref: None 2024-04-16T05:26:47.352 INFO:teuthology.packaging:tag: None 2024-04-16T05:26:47.353 INFO:teuthology.packaging:branch: wip-64502-quincy-3 2024-04-16T05:26:47.353 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T05:26:47.353 DEBUG:teuthology.orchestra.run.smithi122:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/wip-64502-quincy-3/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T05:26:47.439 DEBUG:teuthology.orchestra.run.smithi027:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T05:26:47.488 DEBUG:teuthology.orchestra.run.smithi122:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T05:26:47.585 DEBUG:teuthology.orchestra.run.smithi027:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T05:26:47.615 DEBUG:teuthology.orchestra.run.smithi122:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T05:26:47.690 INFO:teuthology.orchestra.run.smithi027.stdout:check_obsoletes = 1 2024-04-16T05:26:47.695 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum clean all 2024-04-16T05:26:47.700 INFO:teuthology.orchestra.run.smithi122.stdout:check_obsoletes = 1 2024-04-16T05:26:47.704 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum clean all 2024-04-16T05:26:48.049 INFO:teuthology.orchestra.run.smithi122.stdout:74 files removed 2024-04-16T05:26:48.081 INFO:teuthology.orchestra.run.smithi027.stdout:74 files removed 2024-04-16T05:26:48.088 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-radosgw 2024-04-16T05:26:48.122 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-radosgw 2024-04-16T05:26:49.051 INFO:teuthology.orchestra.run.smithi027.stdout:ceph packages for x86_64 210 kB/s | 77 kB 00:00 2024-04-16T05:26:49.275 INFO:teuthology.orchestra.run.smithi122.stdout:ceph packages for x86_64 114 kB/s | 77 kB 00:00 2024-04-16T05:26:49.543 INFO:teuthology.orchestra.run.smithi027.stdout:ceph noarch packages 27 kB/s | 11 kB 00:00 2024-04-16T05:26:49.842 INFO:teuthology.orchestra.run.smithi122.stdout:ceph noarch packages 22 kB/s | 11 kB 00:00 2024-04-16T05:26:50.020 INFO:teuthology.orchestra.run.smithi027.stdout:ceph source packages 4.5 kB/s | 1.8 kB 00:00 2024-04-16T05:26:50.418 INFO:teuthology.orchestra.run.smithi122.stdout:ceph source packages 3.6 kB/s | 1.8 kB 00:00 2024-04-16T05:26:50.714 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-04-16T05:26:51.046 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-04-16T05:26:52.519 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-04-16T05:26:52.789 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - BaseOS 4.5 MB/s | 2.0 MB 00:00 2024-04-16T05:26:53.827 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-16T05:26:54.761 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.0 MB 00:01 2024-04-16T05:26:57.736 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 7.1 MB/s | 19 MB 00:02 2024-04-16T05:26:58.770 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 7.0 MB/s | 19 MB 00:02 2024-04-16T05:27:02.686 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 5.9 MB/s | 6.2 MB 00:01 2024-04-16T05:27:03.404 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-04-16T05:27:04.462 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-04-16T05:27:05.072 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 16 kB 00:00 2024-04-16T05:27:07.024 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-16T05:27:13.249 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-16T05:27:16.414 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:27:16.428 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:27:16.428 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-04-16T05:27:16.428 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:27:16.428 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-16T05:27:16.428 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw x86_64 2:17.2.7-847.g45317f95.el9 ceph 14 M 2024-04-16T05:27:16.428 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2024-04-16T05:27:16.429 INFO:teuthology.orchestra.run.smithi027.stdout: librados2 x86_64 2:17.2.7-847.g45317f95.el9 ceph 3.1 M 2024-04-16T05:27:16.429 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1 x86_64 2:17.2.7-847.g45317f95.el9 ceph 2.9 M 2024-04-16T05:27:16.429 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-04-16T05:27:16.429 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base x86_64 2:17.2.7-847.g45317f95.el9 ceph 5.0 M 2024-04-16T05:27:16.429 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common x86_64 2:17.2.7-847.g45317f95.el9 ceph 22 M 2024-04-16T05:27:16.429 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux x86_64 2:17.2.7-847.g45317f95.el9 ceph 25 k 2024-04-16T05:27:16.429 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T05:27:16.429 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2 x86_64 2:17.2.7-847.g45317f95.el9 ceph 639 k 2024-04-16T05:27:16.429 INFO:teuthology.orchestra.run.smithi027.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T05:27:16.429 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T05:27:16.429 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T05:27:16.429 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1 x86_64 2:17.2.7-847.g45317f95.el9 ceph 457 k 2024-04-16T05:27:16.429 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T05:27:16.430 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2 x86_64 2:17.2.7-847.g45317f95.el9 ceph 4.9 M 2024-04-16T05:27:16.430 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T05:27:16.430 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T05:27:16.430 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse x86_64 2:17.2.7-847.g45317f95.el9 ceph 45 k 2024-04-16T05:27:16.430 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common x86_64 2:17.2.7-847.g45317f95.el9 ceph 117 k 2024-04-16T05:27:16.430 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs x86_64 2:17.2.7-847.g45317f95.el9 ceph 156 k 2024-04-16T05:27:16.430 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T05:27:16.430 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados x86_64 2:17.2.7-847.g45317f95.el9 ceph 320 k 2024-04-16T05:27:16.430 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd x86_64 2:17.2.7-847.g45317f95.el9 ceph 293 k 2024-04-16T05:27:16.430 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw x86_64 2:17.2.7-847.g45317f95.el9 ceph 100 k 2024-04-16T05:27:16.430 INFO:teuthology.orchestra.run.smithi027.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T05:27:16.430 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:27:16.431 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-16T05:27:16.431 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:27:16.431 INFO:teuthology.orchestra.run.smithi027.stdout:Install 22 Packages 2024-04-16T05:27:16.431 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 2 Packages 2024-04-16T05:27:16.431 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:27:16.431 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 58 M 2024-04-16T05:27:16.431 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-16T05:27:17.684 INFO:teuthology.orchestra.run.smithi027.stdout:(1/24): ceph-radosgw-17.2.7-847.g45317f95.el9.x 13 MB/s | 14 MB 00:01 2024-04-16T05:27:17.742 INFO:teuthology.orchestra.run.smithi027.stdout:(2/24): ceph-selinux-17.2.7-847.g45317f95.el9.x 441 kB/s | 25 kB 00:00 2024-04-16T05:27:17.834 INFO:teuthology.orchestra.run.smithi027.stdout:(3/24): libcephfs2-17.2.7-847.g45317f95.el9.x86 6.9 MB/s | 639 kB 00:00 2024-04-16T05:27:17.900 INFO:teuthology.orchestra.run.smithi027.stdout:(4/24): libradosstriper1-17.2.7-847.g45317f95.e 6.8 MB/s | 457 kB 00:00 2024-04-16T05:27:18.226 INFO:teuthology.orchestra.run.smithi027.stdout:(5/24): librgw2-17.2.7-847.g45317f95.el9.x86_64 15 MB/s | 4.9 MB 00:00 2024-04-16T05:27:18.276 INFO:teuthology.orchestra.run.smithi027.stdout:(6/24): python3-ceph-argparse-17.2.7-847.g45317 920 kB/s | 45 kB 00:00 2024-04-16T05:27:18.326 INFO:teuthology.orchestra.run.smithi027.stdout:(7/24): python3-ceph-common-17.2.7-847.g45317f9 2.3 MB/s | 117 kB 00:00 2024-04-16T05:27:18.443 INFO:teuthology.orchestra.run.smithi027.stdout:(8/24): ceph-base-17.2.7-847.g45317f95.el9.x86_ 2.7 MB/s | 5.0 MB 00:01 2024-04-16T05:27:18.477 INFO:teuthology.orchestra.run.smithi027.stdout:(9/24): python3-cephfs-17.2.7-847.g45317f95.el9 1.0 MB/s | 156 kB 00:00 2024-04-16T05:27:18.536 INFO:teuthology.orchestra.run.smithi027.stdout:(10/24): python3-rbd-17.2.7-847.g45317f95.el9.x 5.0 MB/s | 293 kB 00:00 2024-04-16T05:27:18.569 INFO:teuthology.orchestra.run.smithi027.stdout:(11/24): python3-rados-17.2.7-847.g45317f95.el9 2.5 MB/s | 320 kB 00:00 2024-04-16T05:27:18.594 INFO:teuthology.orchestra.run.smithi027.stdout:(12/24): python3-rgw-17.2.7-847.g45317f95.el9.x 1.7 MB/s | 100 kB 00:00 2024-04-16T05:27:18.787 INFO:teuthology.orchestra.run.smithi027.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 365 kB/s | 44 kB 00:00 2024-04-16T05:27:18.812 INFO:teuthology.orchestra.run.smithi027.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 660 kB/s | 159 kB 00:00 2024-04-16T05:27:18.879 INFO:teuthology.orchestra.run.smithi027.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 7.1 MB/s | 661 kB 00:00 2024-04-16T05:27:18.912 INFO:teuthology.orchestra.run.smithi027.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 282 kB 00:00 2024-04-16T05:27:18.937 INFO:teuthology.orchestra.run.smithi027.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 749 kB/s | 41 kB 00:00 2024-04-16T05:27:19.030 INFO:teuthology.orchestra.run.smithi027.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 536 kB/s | 49 kB 00:00 2024-04-16T05:27:19.071 INFO:teuthology.orchestra.run.smithi027.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-04-16T05:27:19.114 INFO:teuthology.orchestra.run.smithi027.stdout:(20/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.5 MB/s | 309 kB 00:00 2024-04-16T05:27:19.222 INFO:teuthology.orchestra.run.smithi027.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-16T05:27:19.548 INFO:teuthology.orchestra.run.smithi027.stdout:(22/24): librbd1-17.2.7-847.g45317f95.el9.x86_6 9.1 MB/s | 2.9 MB 00:00 2024-04-16T05:27:19.748 INFO:teuthology.orchestra.run.smithi027.stdout:(23/24): librados2-17.2.7-847.g45317f95.el9.x86 4.9 MB/s | 3.1 MB 00:00 2024-04-16T05:27:20.156 INFO:teuthology.orchestra.run.smithi027.stdout:(24/24): ceph-common-17.2.7-847.g45317f95.el9.x 6.3 MB/s | 22 MB 00:03 2024-04-16T05:27:20.166 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:27:20.166 INFO:teuthology.orchestra.run.smithi027.stdout:Total 15 MB/s | 58 MB 00:03 2024-04-16T05:27:20.167 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-16T05:27:20.248 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-16T05:27:20.248 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-16T05:27:20.794 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-16T05:27:20.795 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-16T05:27:22.196 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-16T05:27:22.594 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-04-16T05:27:22.650 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librados2-2:17.2.7-847.g45317f95.el9.x86_64 2/26 2024-04-16T05:27:22.785 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:17.2.7-847.g45317f95.el9.x86_64 2/26 2024-04-16T05:27:22.891 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 3/26 2024-04-16T05:27:22.918 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 4/26 2024-04-16T05:27:23.085 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 4/26 2024-04-16T05:27:23.150 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-04-16T05:27:23.192 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-04-16T05:27:23.290 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 21 MB 00:18 2024-04-16T05:27:23.772 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-04-16T05:27:23.804 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librgw2-2:17.2.7-847.g45317f95.el9.x86_64 8/26 2024-04-16T05:27:23.896 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librgw2-2:17.2.7-847.g45317f95.el9.x86_64 8/26 2024-04-16T05:27:23.936 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-04-16T05:27:24.009 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-argparse-2:17.2.7-847.g45317f95.el9.x 10/26 2024-04-16T05:27:24.070 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 11/26 2024-04-16T05:27:24.184 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 12/26 2024-04-16T05:27:24.224 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 13/26 2024-04-16T05:27:24.340 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 13/26 2024-04-16T05:27:24.422 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-04-16T05:27:24.471 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-04-16T05:27:24.519 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-04-16T05:27:24.921 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-04-16T05:27:24.952 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librbd1-2:17.2.7-847.g45317f95.el9.x86_64 18/26 2024-04-16T05:27:25.101 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:17.2.7-847.g45317f95.el9.x86_64 18/26 2024-04-16T05:27:25.200 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 19/26 2024-04-16T05:27:25.230 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-common-2:17.2.7-847.g45317f95.el9.x86 20/26 2024-04-16T05:27:28.199 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 21/26 2024-04-16T05:27:28.243 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 21/26 2024-04-16T05:27:28.877 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 21/26 2024-04-16T05:27:28.922 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 22/26 2024-04-16T05:27:29.086 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 22/26 2024-04-16T05:27:29.086 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T05:27:29.086 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T05:27:29.086 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:27:29.131 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 23/26 2024-04-16T05:27:29.195 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-16T05:27:32.247 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:27:32.258 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:27:32.258 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-16T05:27:32.259 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:27:32.259 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-16T05:27:32.259 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw x86_64 2:17.2.7-847.g45317f95.el9 ceph 14 M 2024-04-16T05:27:32.259 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrading: 2024-04-16T05:27:32.259 INFO:teuthology.orchestra.run.smithi122.stdout: librados2 x86_64 2:17.2.7-847.g45317f95.el9 ceph 3.1 M 2024-04-16T05:27:32.259 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1 x86_64 2:17.2.7-847.g45317f95.el9 ceph 2.9 M 2024-04-16T05:27:32.259 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-16T05:27:32.259 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base x86_64 2:17.2.7-847.g45317f95.el9 ceph 5.0 M 2024-04-16T05:27:32.259 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common x86_64 2:17.2.7-847.g45317f95.el9 ceph 22 M 2024-04-16T05:27:32.259 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux x86_64 2:17.2.7-847.g45317f95.el9 ceph 25 k 2024-04-16T05:27:32.259 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T05:27:32.259 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2 x86_64 2:17.2.7-847.g45317f95.el9 ceph 639 k 2024-04-16T05:27:32.259 INFO:teuthology.orchestra.run.smithi122.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T05:27:32.260 INFO:teuthology.orchestra.run.smithi122.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T05:27:32.260 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T05:27:32.260 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1 x86_64 2:17.2.7-847.g45317f95.el9 ceph 457 k 2024-04-16T05:27:32.260 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T05:27:32.260 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2 x86_64 2:17.2.7-847.g45317f95.el9 ceph 4.9 M 2024-04-16T05:27:32.260 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T05:27:32.260 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T05:27:32.260 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse x86_64 2:17.2.7-847.g45317f95.el9 ceph 45 k 2024-04-16T05:27:32.260 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common x86_64 2:17.2.7-847.g45317f95.el9 ceph 117 k 2024-04-16T05:27:32.260 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs x86_64 2:17.2.7-847.g45317f95.el9 ceph 156 k 2024-04-16T05:27:32.260 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T05:27:32.260 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados x86_64 2:17.2.7-847.g45317f95.el9 ceph 320 k 2024-04-16T05:27:32.260 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd x86_64 2:17.2.7-847.g45317f95.el9 ceph 293 k 2024-04-16T05:27:32.260 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw x86_64 2:17.2.7-847.g45317f95.el9 ceph 100 k 2024-04-16T05:27:32.260 INFO:teuthology.orchestra.run.smithi122.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T05:27:32.260 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:27:32.261 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-16T05:27:32.261 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:27:32.261 INFO:teuthology.orchestra.run.smithi122.stdout:Install 22 Packages 2024-04-16T05:27:32.261 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrade 2 Packages 2024-04-16T05:27:32.261 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:27:32.261 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 58 M 2024-04-16T05:27:32.261 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-16T05:27:34.205 INFO:teuthology.orchestra.run.smithi122.stdout:(1/24): ceph-common-17.2.7-847.g45317f95.el9.x8 13 MB/s | 22 MB 00:01 2024-04-16T05:27:34.255 INFO:teuthology.orchestra.run.smithi122.stdout:(2/24): ceph-selinux-17.2.7-847.g45317f95.el9.x 515 kB/s | 25 kB 00:00 2024-04-16T05:27:34.347 INFO:teuthology.orchestra.run.smithi122.stdout:(3/24): libcephfs2-17.2.7-847.g45317f95.el9.x86 6.9 MB/s | 639 kB 00:00 2024-04-16T05:27:34.422 INFO:teuthology.orchestra.run.smithi122.stdout:(4/24): libradosstriper1-17.2.7-847.g45317f95.e 6.0 MB/s | 457 kB 00:00 2024-04-16T05:27:34.614 INFO:teuthology.orchestra.run.smithi122.stdout:(5/24): ceph-radosgw-17.2.7-847.g45317f95.el9.x 6.6 MB/s | 14 MB 00:02 2024-04-16T05:27:34.673 INFO:teuthology.orchestra.run.smithi122.stdout:(6/24): python3-ceph-argparse-17.2.7-847.g45317 792 kB/s | 45 kB 00:00 2024-04-16T05:27:34.740 INFO:teuthology.orchestra.run.smithi122.stdout:(7/24): python3-ceph-common-17.2.7-847.g45317f9 1.7 MB/s | 117 kB 00:00 2024-04-16T05:27:34.840 INFO:teuthology.orchestra.run.smithi122.stdout:(8/24): librgw2-17.2.7-847.g45317f95.el9.x86_64 12 MB/s | 4.9 MB 00:00 2024-04-16T05:27:34.865 INFO:teuthology.orchestra.run.smithi122.stdout:(9/24): python3-cephfs-17.2.7-847.g45317f95.el9 1.2 MB/s | 156 kB 00:00 2024-04-16T05:27:34.924 INFO:teuthology.orchestra.run.smithi122.stdout:(10/24): python3-rados-17.2.7-847.g45317f95.el9 3.8 MB/s | 320 kB 00:00 2024-04-16T05:27:34.957 INFO:teuthology.orchestra.run.smithi122.stdout:(11/24): python3-rbd-17.2.7-847.g45317f95.el9.x 3.1 MB/s | 293 kB 00:00 2024-04-16T05:27:34.983 INFO:teuthology.orchestra.run.smithi122.stdout:(12/24): python3-rgw-17.2.7-847.g45317f95.el9.x 1.7 MB/s | 100 kB 00:00 2024-04-16T05:27:35.175 INFO:teuthology.orchestra.run.smithi122.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 358 kB/s | 44 kB 00:00 2024-04-16T05:27:35.200 INFO:teuthology.orchestra.run.smithi122.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 660 kB/s | 159 kB 00:00 2024-04-16T05:27:35.258 INFO:teuthology.orchestra.run.smithi122.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 7.9 MB/s | 661 kB 00:00 2024-04-16T05:27:35.292 INFO:teuthology.orchestra.run.smithi122.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 282 kB 00:00 2024-04-16T05:27:35.400 INFO:teuthology.orchestra.run.smithi122.stdout:(17/24): ceph-base-17.2.7-847.g45317f95.el9.x86 1.7 MB/s | 5.0 MB 00:02 2024-04-16T05:27:35.426 INFO:teuthology.orchestra.run.smithi122.stdout:(18/24): python3-prettytable-0.7.2-27.el9.noarc 249 kB/s | 41 kB 00:00 2024-04-16T05:27:35.976 INFO:teuthology.orchestra.run.smithi122.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 88 kB/s | 49 kB 00:00 2024-04-16T05:27:36.001 INFO:teuthology.orchestra.run.smithi122.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 117 kB/s | 67 kB 00:00 2024-04-16T05:27:36.393 INFO:teuthology.orchestra.run.smithi122.stdout:(21/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 281 kB/s | 309 kB 00:01 2024-04-16T05:27:36.685 INFO:teuthology.orchestra.run.smithi122.stdout:(22/24): librbd1-17.2.7-847.g45317f95.el9.x86_6 10 MB/s | 2.9 MB 00:00 2024-04-16T05:27:36.827 INFO:teuthology.orchestra.run.smithi122.stdout:(23/24): thrift-0.15.0-2.el9.x86_64.rpm 1.9 MB/s | 1.6 MB 00:00 2024-04-16T05:27:37.018 INFO:teuthology.orchestra.run.smithi122.stdout:(24/24): librados2-17.2.7-847.g45317f95.el9.x86 3.0 MB/s | 3.1 MB 00:01 2024-04-16T05:27:37.027 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:27:37.028 INFO:teuthology.orchestra.run.smithi122.stdout:Total 12 MB/s | 58 MB 00:04 2024-04-16T05:27:37.029 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-16T05:27:37.102 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-16T05:27:37.102 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-16T05:27:37.640 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-16T05:27:37.640 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-16T05:27:39.001 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-16T05:27:39.392 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-04-16T05:27:39.514 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librados2-2:17.2.7-847.g45317f95.el9.x86_64 2/26 2024-04-16T05:27:39.645 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:17.2.7-847.g45317f95.el9.x86_64 2/26 2024-04-16T05:27:39.754 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 3/26 2024-04-16T05:27:39.781 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 4/26 2024-04-16T05:27:39.929 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 4/26 2024-04-16T05:27:39.988 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-04-16T05:27:40.029 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-04-16T05:27:40.540 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-04-16T05:27:40.560 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 23/26 2024-04-16T05:27:40.560 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /sys 2024-04-16T05:27:40.561 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /proc 2024-04-16T05:27:40.561 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /mnt 2024-04-16T05:27:40.561 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /var/tmp 2024-04-16T05:27:40.561 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /home 2024-04-16T05:27:40.561 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /root 2024-04-16T05:27:40.561 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /tmp 2024-04-16T05:27:40.561 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:27:40.584 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librgw2-2:17.2.7-847.g45317f95.el9.x86_64 8/26 2024-04-16T05:27:40.675 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librgw2-2:17.2.7-847.g45317f95.el9.x86_64 8/26 2024-04-16T05:27:40.710 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-04-16T05:27:40.773 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-argparse-2:17.2.7-847.g45317f95.el9.x 10/26 2024-04-16T05:27:40.829 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 11/26 2024-04-16T05:27:40.960 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 12/26 2024-04-16T05:27:40.986 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 13/26 2024-04-16T05:27:41.088 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 13/26 2024-04-16T05:27:41.181 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-04-16T05:27:41.239 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-04-16T05:27:41.274 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-04-16T05:27:41.660 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-04-16T05:27:41.690 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librbd1-2:17.2.7-847.g45317f95.el9.x86_64 18/26 2024-04-16T05:27:41.804 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:17.2.7-847.g45317f95.el9.x86_64 18/26 2024-04-16T05:27:41.899 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 19/26 2024-04-16T05:27:41.942 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-common-2:17.2.7-847.g45317f95.el9.x86 20/26 2024-04-16T05:27:42.020 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 24/26 2024-04-16T05:27:42.181 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 24/26 2024-04-16T05:27:42.181 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T05:27:42.181 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T05:27:42.181 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T05:27:42.181 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T05:27:42.181 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:27:42.187 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-04-16T05:27:42.317 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-04-16T05:27:42.317 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-04-16T05:27:44.002 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-04-16T05:27:44.002 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 1/26 2024-04-16T05:27:44.002 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 2/26 2024-04-16T05:27:44.002 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 3/26 2024-04-16T05:27:44.002 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 4/26 2024-04-16T05:27:44.002 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 5/26 2024-04-16T05:27:44.002 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 6/26 2024-04-16T05:27:44.003 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librgw2-2:17.2.7-847.g45317f95.el9.x86_64 7/26 2024-04-16T05:27:44.003 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-argparse-2:17.2.7-847.g45317f95.el9.x 8/26 2024-04-16T05:27:44.003 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-common-2:17.2.7-847.g45317f95.el9.x86 9/26 2024-04-16T05:27:44.003 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 10/26 2024-04-16T05:27:44.003 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 11/26 2024-04-16T05:27:44.003 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 12/26 2024-04-16T05:27:44.003 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 13/26 2024-04-16T05:27:44.003 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-04-16T05:27:44.003 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-04-16T05:27:44.003 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-04-16T05:27:44.003 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-04-16T05:27:44.003 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-04-16T05:27:44.003 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-04-16T05:27:44.003 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-04-16T05:27:44.004 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-04-16T05:27:44.004 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-04-16T05:27:44.004 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:17.2.7-847.g45317f95.el9.x86_64 23/26 2024-04-16T05:27:44.004 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-04-16T05:27:44.004 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:17.2.7-847.g45317f95.el9.x86_64 25/26 2024-04-16T05:27:44.774 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 21/26 2024-04-16T05:27:44.775 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-04-16T05:27:44.775 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:27:44.775 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2024-04-16T05:27:44.775 INFO:teuthology.orchestra.run.smithi027.stdout: librados2-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:27:44.775 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:27:44.775 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-16T05:27:44.775 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:27:44.775 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:27:44.775 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:27:44.776 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:27:44.776 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T05:27:44.776 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:27:44.776 INFO:teuthology.orchestra.run.smithi027.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T05:27:44.776 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T05:27:44.776 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T05:27:44.776 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:27:44.776 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T05:27:44.776 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:27:44.776 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T05:27:44.776 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T05:27:44.776 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:27:44.776 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:27:44.776 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:27:44.776 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T05:27:44.777 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:27:44.777 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:27:44.777 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:27:44.777 INFO:teuthology.orchestra.run.smithi027.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T05:27:44.777 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:27:44.777 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:27:44.822 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 21/26 2024-04-16T05:27:45.229 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-test 2024-04-16T05:27:45.433 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 21/26 2024-04-16T05:27:45.476 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 22/26 2024-04-16T05:27:45.624 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 22/26 2024-04-16T05:27:45.624 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T05:27:45.624 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T05:27:45.624 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:27:45.669 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 23/26 2024-04-16T05:27:46.130 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:33 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:27:46.334 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:27:46.335 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:27:46.335 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-04-16T05:27:46.336 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:27:46.336 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-16T05:27:46.336 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test x86_64 2:17.2.7-847.g45317f95.el9 ceph 48 M 2024-04-16T05:27:46.336 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-04-16T05:27:46.336 INFO:teuthology.orchestra.run.smithi027.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T05:27:46.336 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite x86_64 2:17.2.7-847.g45317f95.el9 ceph 163 k 2024-04-16T05:27:46.336 INFO:teuthology.orchestra.run.smithi027.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T05:27:46.336 INFO:teuthology.orchestra.run.smithi027.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T05:27:46.336 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T05:27:46.336 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:27:46.336 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-16T05:27:46.336 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:27:46.336 INFO:teuthology.orchestra.run.smithi027.stdout:Install 6 Packages 2024-04-16T05:27:46.337 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:27:46.337 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 49 M 2024-04-16T05:27:46.337 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 193 M 2024-04-16T05:27:46.337 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-16T05:27:46.603 INFO:teuthology.orchestra.run.smithi027.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-04-16T05:27:46.653 INFO:teuthology.orchestra.run.smithi027.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 300 kB 00:00 2024-04-16T05:27:46.678 INFO:teuthology.orchestra.run.smithi027.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-04-16T05:27:46.837 INFO:teuthology.orchestra.run.smithi027.stdout:(4/6): libcephsqlite-17.2.7-847.g45317f95.el9.x 381 kB/s | 163 kB 00:00 2024-04-16T05:27:47.587 INFO:teuthology.orchestra.run.smithi027.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 206 kB/s | 187 kB 00:00 2024-04-16T05:27:49.946 INFO:teuthology.orchestra.run.smithi027.stdout:(6/6): ceph-test-17.2.7-847.g45317f95.el9.x86_6 14 MB/s | 48 MB 00:03 2024-04-16T05:27:49.952 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:27:49.952 INFO:teuthology.orchestra.run.smithi027.stdout:Total 13 MB/s | 49 MB 00:03 2024-04-16T05:27:49.996 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-16T05:27:50.054 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-16T05:27:50.054 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-16T05:27:50.520 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-16T05:27:50.520 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-16T05:27:51.221 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-16T05:27:51.324 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-16T05:27:51.371 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-16T05:27:51.418 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T05:27:51.481 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-16T05:27:51.507 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 5/6 2024-04-16T05:27:56.595 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 23/26 2024-04-16T05:27:56.595 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /sys 2024-04-16T05:27:56.595 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /proc 2024-04-16T05:27:56.595 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /mnt 2024-04-16T05:27:56.595 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /var/tmp 2024-04-16T05:27:56.595 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /home 2024-04-16T05:27:56.595 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /root 2024-04-16T05:27:56.595 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /tmp 2024-04-16T05:27:56.595 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:27:56.650 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 5/6 2024-04-16T05:27:56.766 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 6/6 2024-04-16T05:27:58.024 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 24/26 2024-04-16T05:27:58.150 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 24/26 2024-04-16T05:27:58.150 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T05:27:58.150 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T05:27:58.150 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T05:27:58.150 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T05:27:58.151 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:27:58.155 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-04-16T05:27:58.279 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-04-16T05:27:58.280 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-04-16T05:27:59.311 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 6/6 2024-04-16T05:27:59.311 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 1/6 2024-04-16T05:27:59.311 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 2/6 2024-04-16T05:27:59.311 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T05:27:59.311 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-04-16T05:27:59.311 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-04-16T05:27:59.664 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-04-16T05:27:59.664 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:27:59.664 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-16T05:27:59.665 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:27:59.665 INFO:teuthology.orchestra.run.smithi027.stdout: jq-1.6-16.el9.x86_64 2024-04-16T05:27:59.665 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:27:59.665 INFO:teuthology.orchestra.run.smithi027.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T05:27:59.665 INFO:teuthology.orchestra.run.smithi027.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T05:27:59.665 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T05:27:59.665 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:27:59.665 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:28:00.011 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph 2024-04-16T05:28:00.014 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-04-16T05:28:00.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 1/26 2024-04-16T05:28:00.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 2/26 2024-04-16T05:28:00.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 3/26 2024-04-16T05:28:00.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 4/26 2024-04-16T05:28:00.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 5/26 2024-04-16T05:28:00.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 6/26 2024-04-16T05:28:00.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librgw2-2:17.2.7-847.g45317f95.el9.x86_64 7/26 2024-04-16T05:28:00.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-argparse-2:17.2.7-847.g45317f95.el9.x 8/26 2024-04-16T05:28:00.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-common-2:17.2.7-847.g45317f95.el9.x86 9/26 2024-04-16T05:28:00.014 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 10/26 2024-04-16T05:28:00.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 11/26 2024-04-16T05:28:00.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 12/26 2024-04-16T05:28:00.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 13/26 2024-04-16T05:28:00.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-04-16T05:28:00.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-04-16T05:28:00.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-04-16T05:28:00.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-04-16T05:28:00.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-04-16T05:28:00.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-04-16T05:28:00.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-04-16T05:28:00.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-04-16T05:28:00.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-04-16T05:28:00.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:17.2.7-847.g45317f95.el9.x86_64 23/26 2024-04-16T05:28:00.015 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-04-16T05:28:00.016 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:17.2.7-847.g45317f95.el9.x86_64 25/26 2024-04-16T05:28:00.864 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-04-16T05:28:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:00.865 INFO:teuthology.orchestra.run.smithi122.stdout:Upgraded: 2024-04-16T05:28:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: librados2-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:00.865 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-16T05:28:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T05:28:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T05:28:00.865 INFO:teuthology.orchestra.run.smithi122.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T05:28:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T05:28:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T05:28:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T05:28:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T05:28:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T05:28:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:00.866 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:00.867 INFO:teuthology.orchestra.run.smithi122.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T05:28:00.867 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:00.867 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:28:00.927 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:47 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:28:01.097 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:28:01.100 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================== 2024-04-16T05:28:01.100 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-04-16T05:28:01.100 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================== 2024-04-16T05:28:01.101 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-16T05:28:01.101 INFO:teuthology.orchestra.run.smithi027.stdout: ceph x86_64 2:17.2.7-847.g45317f95.el9 ceph 6.5 k 2024-04-16T05:28:01.101 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-04-16T05:28:01.101 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds x86_64 2:17.2.7-847.g45317f95.el9 ceph 2.1 M 2024-04-16T05:28:01.101 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr x86_64 2:17.2.7-847.g45317f95.el9 ceph 1.4 M 2024-04-16T05:28:01.101 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 240 k 2024-04-16T05:28:01.101 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon x86_64 2:17.2.7-847.g45317f95.el9 ceph 3.9 M 2024-04-16T05:28:01.101 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd x86_64 2:17.2.7-847.g45317f95.el9 ceph 15 M 2024-04-16T05:28:01.101 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T05:28:01.101 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T05:28:01.101 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T05:28:01.101 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T05:28:01.102 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T05:28:01.102 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T05:28:01.102 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T05:28:01.102 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T05:28:01.102 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T05:28:01.102 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T05:28:01.102 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T05:28:01.102 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T05:28:01.102 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T05:28:01.102 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T05:28:01.102 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T05:28:01.102 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T05:28:01.103 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T05:28:01.103 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T05:28:01.103 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T05:28:01.103 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T05:28:01.103 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T05:28:01.103 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T05:28:01.103 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T05:28:01.103 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T05:28:01.103 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T05:28:01.103 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T05:28:01.103 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T05:28:01.103 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:01.103 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-16T05:28:01.103 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================== 2024-04-16T05:28:01.104 INFO:teuthology.orchestra.run.smithi027.stdout:Install 33 Packages 2024-04-16T05:28:01.104 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:01.104 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 27 M 2024-04-16T05:28:01.104 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 96 M 2024-04-16T05:28:01.104 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-16T05:28:01.320 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-test 2024-04-16T05:28:01.633 INFO:teuthology.orchestra.run.smithi027.stdout:(1/33): ceph-17.2.7-847.g45317f95.el9.x86_64.rp 22 kB/s | 6.5 kB 00:00 2024-04-16T05:28:01.791 INFO:teuthology.orchestra.run.smithi027.stdout:(2/33): ceph-mds-17.2.7-847.g45317f95.el9.x86_6 4.5 MB/s | 2.1 MB 00:00 2024-04-16T05:28:01.958 INFO:teuthology.orchestra.run.smithi027.stdout:(3/33): ceph-mgr-17.2.7-847.g45317f95.el9.x86_6 2.2 MB/s | 1.4 MB 00:00 2024-04-16T05:28:02.058 INFO:teuthology.orchestra.run.smithi027.stdout:(4/33): ceph-mgr-modules-core-17.2.7-847.g45317 2.4 MB/s | 240 kB 00:00 2024-04-16T05:28:02.166 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:33 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:28:02.200 INFO:teuthology.orchestra.run.smithi027.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 144 kB 00:00 2024-04-16T05:28:02.234 INFO:teuthology.orchestra.run.smithi027.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 967 kB/s | 32 kB 00:00 2024-04-16T05:28:02.267 INFO:teuthology.orchestra.run.smithi027.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-04-16T05:28:02.301 INFO:teuthology.orchestra.run.smithi027.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.2 MB/s | 39 kB 00:00 2024-04-16T05:28:02.343 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:28:02.344 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:28:02.344 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-16T05:28:02.344 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:28:02.344 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-16T05:28:02.344 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test x86_64 2:17.2.7-847.g45317f95.el9 ceph 48 M 2024-04-16T05:28:02.344 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-16T05:28:02.345 INFO:teuthology.orchestra.run.smithi122.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T05:28:02.345 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite x86_64 2:17.2.7-847.g45317f95.el9 ceph 163 k 2024-04-16T05:28:02.345 INFO:teuthology.orchestra.run.smithi122.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T05:28:02.345 INFO:teuthology.orchestra.run.smithi122.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T05:28:02.345 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T05:28:02.345 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:02.345 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-16T05:28:02.345 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:28:02.345 INFO:teuthology.orchestra.run.smithi122.stdout:Install 6 Packages 2024-04-16T05:28:02.345 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:02.346 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 49 M 2024-04-16T05:28:02.346 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 193 M 2024-04-16T05:28:02.346 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-16T05:28:02.351 INFO:teuthology.orchestra.run.smithi027.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.8 MB/s | 241 kB 00:00 2024-04-16T05:28:02.427 INFO:teuthology.orchestra.run.smithi027.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-04-16T05:28:02.460 INFO:teuthology.orchestra.run.smithi027.stdout:(11/33): python3-ply-3.11-14.el9.noarch.rpm 3.1 MB/s | 103 kB 00:00 2024-04-16T05:28:02.502 INFO:teuthology.orchestra.run.smithi027.stdout:(12/33): python3-pycparser-2.20-6.el9.noarch.rp 3.0 MB/s | 124 kB 00:00 2024-04-16T05:28:02.544 INFO:teuthology.orchestra.run.smithi027.stdout:(13/33): python3-urllib3-1.26.5-5.el9.noarch.rp 4.5 MB/s | 187 kB 00:00 2024-04-16T05:28:02.770 INFO:teuthology.orchestra.run.smithi027.stdout:(14/33): ceph-osd-17.2.7-847.g45317f95.el9.x86_ 16 MB/s | 15 MB 00:00 2024-04-16T05:28:02.878 INFO:teuthology.orchestra.run.smithi027.stdout:(15/33): ceph-mon-17.2.7-847.g45317f95.el9.x86_ 3.2 MB/s | 3.9 MB 00:01 2024-04-16T05:28:02.891 INFO:teuthology.orchestra.run.smithi122.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 805 kB/s | 217 kB 00:00 2024-04-16T05:28:02.903 INFO:teuthology.orchestra.run.smithi027.stdout:(16/33): python3-requests-2.25.1-8.el9.noarch.r 349 kB/s | 125 kB 00:00 2024-04-16T05:28:02.948 INFO:teuthology.orchestra.run.smithi122.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.3 MB/s | 300 kB 00:00 2024-04-16T05:28:02.982 INFO:teuthology.orchestra.run.smithi122.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-04-16T05:28:03.040 INFO:teuthology.orchestra.run.smithi122.stdout:(4/6): libcephsqlite-17.2.7-847.g45317f95.el9.x 388 kB/s | 163 kB 00:00 2024-04-16T05:28:03.324 INFO:teuthology.orchestra.run.smithi122.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 549 kB/s | 187 kB 00:00 2024-04-16T05:28:03.404 INFO:teuthology.orchestra.run.smithi027.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 69 kB/s | 43 kB 00:00 2024-04-16T05:28:03.529 INFO:teuthology.orchestra.run.smithi027.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 86 kB/s | 11 kB 00:00 2024-04-16T05:28:03.588 INFO:teuthology.orchestra.run.smithi027.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 242 kB/s | 172 kB 00:00 2024-04-16T05:28:03.663 INFO:teuthology.orchestra.run.smithi027.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 133 kB/s | 18 kB 00:00 2024-04-16T05:28:03.705 INFO:teuthology.orchestra.run.smithi027.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 199 kB/s | 23 kB 00:00 2024-04-16T05:28:03.738 INFO:teuthology.orchestra.run.smithi027.stdout:(22/33): python3-cherrypy-18.6.1-2.el9.noarch.r 469 kB/s | 358 kB 00:00 2024-04-16T05:28:03.797 INFO:teuthology.orchestra.run.smithi027.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 146 kB/s | 19 kB 00:00 2024-04-16T05:28:03.822 INFO:teuthology.orchestra.run.smithi027.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 169 kB/s | 20 kB 00:00 2024-04-16T05:28:03.848 INFO:teuthology.orchestra.run.smithi027.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 427 kB/s | 46 kB 00:00 2024-04-16T05:28:03.956 INFO:teuthology.orchestra.run.smithi027.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 152 kB/s | 16 kB 00:00 2024-04-16T05:28:04.023 INFO:teuthology.orchestra.run.smithi027.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-04-16T05:28:04.057 INFO:teuthology.orchestra.run.smithi027.stdout:(28/33): python3-more-itertools-8.12.0-2.el9.no 305 kB/s | 79 kB 00:00 2024-04-16T05:28:04.082 INFO:teuthology.orchestra.run.smithi027.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 721 kB/s | 90 kB 00:00 2024-04-16T05:28:04.149 INFO:teuthology.orchestra.run.smithi027.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 286 kB/s | 36 kB 00:00 2024-04-16T05:28:04.191 INFO:teuthology.orchestra.run.smithi027.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-04-16T05:28:04.266 INFO:teuthology.orchestra.run.smithi027.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.3 MB/s | 427 kB 00:00 2024-04-16T05:28:04.291 INFO:teuthology.orchestra.run.smithi027.stdout:(33/33): python3-zc-lockfile-2.0-10.el9.noarch. 141 kB/s | 20 kB 00:00 2024-04-16T05:28:04.305 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:28:04.305 INFO:teuthology.orchestra.run.smithi027.stdout:Total 8.4 MB/s | 27 MB 00:03 2024-04-16T05:28:04.349 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-16T05:28:04.442 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-16T05:28:04.442 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-16T05:28:04.870 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-16T05:28:04.871 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-16T05:28:05.141 INFO:teuthology.orchestra.run.smithi122.stdout:(6/6): ceph-test-17.2.7-847.g45317f95.el9.x86_6 19 MB/s | 48 MB 00:02 2024-04-16T05:28:05.147 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:28:05.148 INFO:teuthology.orchestra.run.smithi122.stdout:Total 17 MB/s | 49 MB 00:02 2024-04-16T05:28:05.189 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-16T05:28:05.244 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-16T05:28:05.244 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-16T05:28:05.707 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-16T05:28:05.708 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-16T05:28:06.285 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-16T05:28:06.354 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-16T05:28:06.390 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-16T05:28:06.391 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-16T05:28:06.430 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-16T05:28:06.486 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-04-16T05:28:06.530 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-04-16T05:28:06.574 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T05:28:06.580 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-16T05:28:06.631 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-04-16T05:28:06.661 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 5/6 2024-04-16T05:28:06.837 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-16T05:28:06.905 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-16T05:28:07.015 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-16T05:28:07.083 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-04-16T05:28:07.134 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-04-16T05:28:07.212 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-04-16T05:28:07.307 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-04-16T05:28:07.583 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-04-16T05:28:07.645 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-04-16T05:28:07.688 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-04-16T05:28:07.736 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-04-16T05:28:07.773 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-16T05:28:07.897 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-04-16T05:28:07.939 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-16T05:28:07.972 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-04-16T05:28:08.015 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-04-16T05:28:08.050 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-04-16T05:28:08.084 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-04-16T05:28:08.215 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-04-16T05:28:08.264 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-04-16T05:28:08.380 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-04-16T05:28:08.569 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-04-16T05:28:08.734 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-16T05:28:08.912 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-847.g45317f95.el9.n 28/33 2024-04-16T05:28:08.940 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 29/33 2024-04-16T05:28:09.019 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 29/33 2024-04-16T05:28:09.019 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T05:28:09.019 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T05:28:09.019 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T05:28:09.019 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T05:28:09.019 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:10.541 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 30/33 2024-04-16T05:28:10.627 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 30/33 2024-04-16T05:28:10.628 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T05:28:10.628 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T05:28:10.628 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T05:28:10.628 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T05:28:10.628 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:11.058 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 31/33 2024-04-16T05:28:11.133 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 31/33 2024-04-16T05:28:11.133 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T05:28:11.133 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T05:28:11.133 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T05:28:11.133 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T05:28:11.134 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:11.367 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 32/33 2024-04-16T05:28:11.441 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 32/33 2024-04-16T05:28:11.441 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T05:28:11.441 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T05:28:11.442 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T05:28:11.442 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T05:28:11.442 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:11.496 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 5/6 2024-04-16T05:28:11.509 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-2:17.2.7-847.g45317f95.el9.x86_64 33/33 2024-04-16T05:28:11.590 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 6/6 2024-04-16T05:28:11.623 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-2:17.2.7-847.g45317f95.el9.x86_64 33/33 2024-04-16T05:28:11.623 INFO:teuthology.orchestra.run.smithi027.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-16T05:28:11.623 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:13.503 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-2:17.2.7-847.g45317f95.el9.x86_64 1/33 2024-04-16T05:28:13.503 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 2/33 2024-04-16T05:28:13.503 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 3/33 2024-04-16T05:28:13.503 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 4/33 2024-04-16T05:28:13.503 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 5/33 2024-04-16T05:28:13.503 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-847.g45317f95.el9.n 6/33 2024-04-16T05:28:13.503 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-04-16T05:28:13.503 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-04-16T05:28:13.503 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-04-16T05:28:13.503 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-04-16T05:28:13.503 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-04-16T05:28:13.504 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-04-16T05:28:13.504 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-04-16T05:28:13.504 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-04-16T05:28:13.504 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-04-16T05:28:13.504 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-04-16T05:28:13.504 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-16T05:28:13.504 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-16T05:28:13.504 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-16T05:28:13.504 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-16T05:28:13.504 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-16T05:28:13.504 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-16T05:28:13.504 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-16T05:28:13.504 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-16T05:28:13.505 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-16T05:28:13.505 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-16T05:28:13.505 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-16T05:28:13.505 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-16T05:28:13.505 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-16T05:28:13.505 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-16T05:28:13.505 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-16T05:28:13.505 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-16T05:28:13.876 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 6/6 2024-04-16T05:28:13.876 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 1/6 2024-04-16T05:28:13.876 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 2/6 2024-04-16T05:28:13.876 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-04-16T05:28:13.877 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-04-16T05:28:13.877 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-04-16T05:28:14.214 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-04-16T05:28:14.215 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:14.215 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-16T05:28:14.215 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:14.215 INFO:teuthology.orchestra.run.smithi122.stdout: jq-1.6-16.el9.x86_64 2024-04-16T05:28:14.215 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:14.215 INFO:teuthology.orchestra.run.smithi122.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T05:28:14.215 INFO:teuthology.orchestra.run.smithi122.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T05:28:14.215 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T05:28:14.215 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:14.215 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:28:14.574 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph 2024-04-16T05:28:14.982 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-16T05:28:14.982 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:14.982 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-16T05:28:14.982 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:14.982 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:14.982 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:14.982 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T05:28:14.982 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:14.982 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:14.982 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T05:28:14.982 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T05:28:14.983 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T05:28:14.983 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T05:28:14.983 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T05:28:14.983 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T05:28:14.983 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T05:28:14.983 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T05:28:14.983 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T05:28:14.983 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T05:28:14.983 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T05:28:14.983 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T05:28:14.983 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T05:28:14.983 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T05:28:14.983 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T05:28:14.983 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T05:28:14.983 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T05:28:14.983 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T05:28:14.984 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T05:28:14.984 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T05:28:14.984 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T05:28:14.984 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T05:28:14.984 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T05:28:14.984 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T05:28:14.984 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T05:28:14.984 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T05:28:14.984 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T05:28:14.984 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:14.984 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:28:15.381 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:00:46 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:28:15.458 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-base 2024-04-16T05:28:15.548 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:28:15.551 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================== 2024-04-16T05:28:15.551 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-16T05:28:15.551 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================== 2024-04-16T05:28:15.551 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-16T05:28:15.551 INFO:teuthology.orchestra.run.smithi122.stdout: ceph x86_64 2:17.2.7-847.g45317f95.el9 ceph 6.5 k 2024-04-16T05:28:15.551 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-16T05:28:15.552 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds x86_64 2:17.2.7-847.g45317f95.el9 ceph 2.1 M 2024-04-16T05:28:15.552 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr x86_64 2:17.2.7-847.g45317f95.el9 ceph 1.4 M 2024-04-16T05:28:15.552 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 240 k 2024-04-16T05:28:15.552 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon x86_64 2:17.2.7-847.g45317f95.el9 ceph 3.9 M 2024-04-16T05:28:15.552 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd x86_64 2:17.2.7-847.g45317f95.el9 ceph 15 M 2024-04-16T05:28:15.552 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T05:28:15.552 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T05:28:15.552 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T05:28:15.552 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T05:28:15.552 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T05:28:15.552 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T05:28:15.552 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T05:28:15.552 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T05:28:15.552 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T05:28:15.553 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T05:28:15.553 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T05:28:15.553 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T05:28:15.553 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T05:28:15.553 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T05:28:15.553 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T05:28:15.553 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T05:28:15.553 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T05:28:15.553 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T05:28:15.553 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T05:28:15.553 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T05:28:15.554 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T05:28:15.554 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T05:28:15.554 INFO:teuthology.orchestra.run.smithi122.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T05:28:15.554 INFO:teuthology.orchestra.run.smithi122.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T05:28:15.554 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T05:28:15.554 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T05:28:15.554 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T05:28:15.554 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:15.554 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-16T05:28:15.554 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================== 2024-04-16T05:28:15.554 INFO:teuthology.orchestra.run.smithi122.stdout:Install 33 Packages 2024-04-16T05:28:15.554 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:15.555 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 27 M 2024-04-16T05:28:15.555 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 96 M 2024-04-16T05:28:15.555 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-16T05:28:16.382 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:03 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:28:16.427 INFO:teuthology.orchestra.run.smithi122.stdout:(1/33): ceph-17.2.7-847.g45317f95.el9.x86_64.rp 22 kB/s | 6.5 kB 00:00 2024-04-16T05:28:16.488 INFO:teuthology.orchestra.run.smithi027.stdout:Package ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T05:28:16.544 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:28:16.546 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-16T05:28:16.546 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:28:16.612 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install cephadm 2024-04-16T05:28:16.810 INFO:teuthology.orchestra.run.smithi122.stdout:(2/33): ceph-mds-17.2.7-847.g45317f95.el9.x86_6 3.0 MB/s | 2.1 MB 00:00 2024-04-16T05:28:16.861 INFO:teuthology.orchestra.run.smithi122.stdout:(3/33): ceph-mgr-17.2.7-847.g45317f95.el9.x86_6 1.9 MB/s | 1.4 MB 00:00 2024-04-16T05:28:16.994 INFO:teuthology.orchestra.run.smithi122.stdout:(4/33): ceph-mon-17.2.7-847.g45317f95.el9.x86_6 6.9 MB/s | 3.9 MB 00:00 2024-04-16T05:28:17.028 INFO:teuthology.orchestra.run.smithi122.stdout:(5/33): ceph-mgr-modules-core-17.2.7-847.g45317 1.4 MB/s | 240 kB 00:00 2024-04-16T05:28:17.237 INFO:teuthology.orchestra.run.smithi122.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 152 kB/s | 32 kB 00:00 2024-04-16T05:28:17.262 INFO:teuthology.orchestra.run.smithi122.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 542 kB/s | 144 kB 00:00 2024-04-16T05:28:17.287 INFO:teuthology.orchestra.run.smithi122.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 970 kB/s | 47 kB 00:00 2024-04-16T05:28:17.312 INFO:teuthology.orchestra.run.smithi122.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 784 kB/s | 39 kB 00:00 2024-04-16T05:28:17.354 INFO:teuthology.orchestra.run.smithi122.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.6 MB/s | 241 kB 00:00 2024-04-16T05:28:17.430 INFO:teuthology.orchestra.run.smithi122.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 10 MB/s | 1.1 MB 00:00 2024-04-16T05:28:17.455 INFO:teuthology.orchestra.run.smithi122.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2024-04-16T05:28:17.480 INFO:teuthology.orchestra.run.smithi122.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.5 MB/s | 124 kB 00:00 2024-04-16T05:28:17.506 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:04 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:28:17.514 INFO:teuthology.orchestra.run.smithi122.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.2 MB/s | 187 kB 00:00 2024-04-16T05:28:17.631 INFO:teuthology.orchestra.run.smithi122.stdout:(15/33): python3-requests-2.25.1-8.el9.noarch.r 846 kB/s | 125 kB 00:00 2024-04-16T05:28:17.656 INFO:teuthology.orchestra.run.smithi122.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 308 kB/s | 43 kB 00:00 2024-04-16T05:28:17.668 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:28:17.669 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:28:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-04-16T05:28:17.670 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:28:17.670 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-16T05:28:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 82 k 2024-04-16T05:28:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:17.670 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-16T05:28:17.670 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:28:17.670 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-04-16T05:28:17.670 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:17.670 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 82 k 2024-04-16T05:28:17.670 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 372 k 2024-04-16T05:28:17.670 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-16T05:28:17.892 INFO:teuthology.orchestra.run.smithi027.stdout:cephadm-17.2.7-847.g45317f95.el9.noarch.rpm 370 kB/s | 82 kB 00:00 2024-04-16T05:28:17.894 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:28:17.895 INFO:teuthology.orchestra.run.smithi027.stdout:Total 365 kB/s | 82 kB 00:00 2024-04-16T05:28:17.895 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-16T05:28:17.898 INFO:teuthology.orchestra.run.smithi122.stdout:(17/33): ceph-osd-17.2.7-847.g45317f95.el9.x86_ 14 MB/s | 15 MB 00:01 2024-04-16T05:28:17.913 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-16T05:28:17.914 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-16T05:28:17.921 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-16T05:28:17.922 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-16T05:28:17.981 INFO:teuthology.orchestra.run.smithi122.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 130 kB/s | 11 kB 00:00 2024-04-16T05:28:18.055 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-16T05:28:18.066 INFO:teuthology.orchestra.run.smithi122.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 396 kB/s | 172 kB 00:00 2024-04-16T05:28:18.091 INFO:teuthology.orchestra.run.smithi122.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-04-16T05:28:18.124 INFO:teuthology.orchestra.run.smithi122.stdout:(21/33): python3-cherrypy-18.6.1-2.el9.noarch.r 767 kB/s | 358 kB 00:00 2024-04-16T05:28:18.149 INFO:teuthology.orchestra.run.smithi122.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 278 kB/s | 23 kB 00:00 2024-04-16T05:28:18.175 INFO:teuthology.orchestra.run.smithi122.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 234 kB/s | 19 kB 00:00 2024-04-16T05:28:18.200 INFO:teuthology.orchestra.run.smithi122.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 263 kB/s | 20 kB 00:00 2024-04-16T05:28:18.242 INFO:teuthology.orchestra.run.smithi122.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 506 kB/s | 46 kB 00:00 2024-04-16T05:28:18.301 INFO:teuthology.orchestra.run.smithi122.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 632 kB/s | 79 kB 00:00 2024-04-16T05:28:18.326 INFO:teuthology.orchestra.run.smithi122.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 198 kB/s | 16 kB 00:00 2024-04-16T05:28:18.368 INFO:teuthology.orchestra.run.smithi122.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-04-16T05:28:18.393 INFO:teuthology.orchestra.run.smithi122.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 540 kB/s | 36 kB 00:00 2024-04-16T05:28:18.427 INFO:teuthology.orchestra.run.smithi122.stdout:(30/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 719 kB/s | 90 kB 00:00 2024-04-16T05:28:18.494 INFO:teuthology.orchestra.run.smithi122.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 303 kB/s | 20 kB 00:00 2024-04-16T05:28:18.527 INFO:teuthology.orchestra.run.smithi122.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2024-04-16T05:28:18.577 INFO:teuthology.orchestra.run.smithi122.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.3 MB/s | 427 kB 00:00 2024-04-16T05:28:18.585 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:28:18.585 INFO:teuthology.orchestra.run.smithi122.stdout:Total 8.9 MB/s | 27 MB 00:03 2024-04-16T05:28:18.626 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-16T05:28:18.666 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T05:28:18.710 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-16T05:28:18.711 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-16T05:28:18.827 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T05:28:19.105 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-16T05:28:19.106 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-16T05:28:19.338 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T05:28:19.664 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T05:28:19.665 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:19.665 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-16T05:28:19.665 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T05:28:19.665 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:19.665 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:28:19.976 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T05:28:20.485 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-16T05:28:20.539 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-04-16T05:28:20.574 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-04-16T05:28:20.619 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-04-16T05:28:20.755 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-04-16T05:28:20.894 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-04-16T05:28:20.898 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:07 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:28:20.955 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-04-16T05:28:21.059 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-04-16T05:28:21.065 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:28:21.065 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:28:21.065 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-04-16T05:28:21.066 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:28:21.066 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-16T05:28:21.066 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-847.g45317f95.el9 ceph 137 k 2024-04-16T05:28:21.066 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:21.066 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-16T05:28:21.066 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:28:21.066 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-04-16T05:28:21.066 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:21.067 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 137 k 2024-04-16T05:28:21.067 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 379 k 2024-04-16T05:28:21.067 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-16T05:28:21.124 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-04-16T05:28:21.167 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-04-16T05:28:21.231 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-04-16T05:28:21.311 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-04-16T05:28:21.449 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-immutable-object-cache-17.2.7-847.g45317f9 360 kB/s | 137 kB 00:00 2024-04-16T05:28:21.451 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:28:21.451 INFO:teuthology.orchestra.run.smithi027.stdout:Total 357 kB/s | 137 kB 00:00 2024-04-16T05:28:21.452 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-16T05:28:21.479 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-16T05:28:21.479 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-16T05:28:21.557 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-04-16T05:28:21.582 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-16T05:28:21.582 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-16T05:28:21.613 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-04-16T05:28:21.651 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-04-16T05:28:21.696 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-04-16T05:28:21.732 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-04-16T05:28:21.796 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-16T05:28:21.838 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-04-16T05:28:21.858 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-847.g45317f95.e 1/1 2024-04-16T05:28:21.905 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-16T05:28:21.933 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-04-16T05:28:21.937 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-847.g45317f95.e 1/1 2024-04-16T05:28:21.937 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T05:28:21.938 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T05:28:21.938 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:21.968 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-04-16T05:28:22.018 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-04-16T05:28:22.053 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-04-16T05:28:22.162 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-04-16T05:28:22.208 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-04-16T05:28:22.303 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-04-16T05:28:22.472 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-04-16T05:28:22.624 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-16T05:28:22.786 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-847.g45317f95.el9.n 28/33 2024-04-16T05:28:22.818 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 29/33 2024-04-16T05:28:22.887 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 29/33 2024-04-16T05:28:22.888 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T05:28:22.888 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T05:28:22.888 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T05:28:22.888 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T05:28:22.888 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:23.402 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-847.g45317f95.e 1/1 2024-04-16T05:28:23.402 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:23.403 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-16T05:28:23.403 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:23.403 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:23.403 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:28:23.698 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr 2024-04-16T05:28:24.294 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 30/33 2024-04-16T05:28:24.373 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 30/33 2024-04-16T05:28:24.373 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T05:28:24.373 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T05:28:24.373 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T05:28:24.373 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T05:28:24.373 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:24.617 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:11 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:28:24.723 INFO:teuthology.orchestra.run.smithi027.stdout:Package ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T05:28:24.781 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:28:24.783 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-16T05:28:24.783 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:28:24.794 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 31/33 2024-04-16T05:28:24.851 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T05:28:24.862 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 31/33 2024-04-16T05:28:24.862 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T05:28:24.862 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T05:28:24.862 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T05:28:24.862 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T05:28:24.862 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:25.078 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 32/33 2024-04-16T05:28:25.152 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 32/33 2024-04-16T05:28:25.152 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T05:28:25.152 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T05:28:25.153 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T05:28:25.153 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T05:28:25.153 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:25.219 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-2:17.2.7-847.g45317f95.el9.x86_64 33/33 2024-04-16T05:28:25.340 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-2:17.2.7-847.g45317f95.el9.x86_64 33/33 2024-04-16T05:28:25.340 INFO:teuthology.orchestra.run.smithi122.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-16T05:28:25.340 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:25.755 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:12 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:28:25.921 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:28:25.922 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:28:25.922 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-04-16T05:28:25.922 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:28:25.922 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-16T05:28:25.922 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 3.4 M 2024-04-16T05:28:25.922 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-04-16T05:28:25.922 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 24 k 2024-04-16T05:28:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 15 k 2024-04-16T05:28:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-16T05:28:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T05:28:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T05:28:25.923 INFO:teuthology.orchestra.run.smithi027.stdout:Installing weak dependencies: 2024-04-16T05:28:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-16T05:28:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:25.923 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-16T05:28:25.923 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:28:25.923 INFO:teuthology.orchestra.run.smithi027.stdout:Install 7 Packages 2024-04-16T05:28:25.923 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:25.924 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 3.7 M 2024-04-16T05:28:25.924 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 80 M 2024-04-16T05:28:25.924 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-16T05:28:26.337 INFO:teuthology.orchestra.run.smithi027.stdout:(1/7): ceph-grafana-dashboards-17.2.7-847.g4531 76 kB/s | 24 kB 00:00 2024-04-16T05:28:26.378 INFO:teuthology.orchestra.run.smithi027.stdout:(2/7): ceph-prometheus-alerts-17.2.7-847.g45317 40 kB/s | 15 kB 00:00 2024-04-16T05:28:26.646 INFO:teuthology.orchestra.run.smithi027.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 29 kB/s | 9.0 kB 00:00 2024-04-16T05:28:26.754 INFO:teuthology.orchestra.run.smithi027.stdout:(4/7): ceph-mgr-dashboard-17.2.7-847.g45317f95. 4.5 MB/s | 3.4 MB 00:00 2024-04-16T05:28:26.779 INFO:teuthology.orchestra.run.smithi027.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 232 kB/s | 31 kB 00:00 2024-04-16T05:28:26.804 INFO:teuthology.orchestra.run.smithi027.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 96 kB/s | 41 kB 00:00 2024-04-16T05:28:27.153 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-2:17.2.7-847.g45317f95.el9.x86_64 1/33 2024-04-16T05:28:27.153 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 2/33 2024-04-16T05:28:27.153 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 3/33 2024-04-16T05:28:27.153 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 4/33 2024-04-16T05:28:27.153 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 5/33 2024-04-16T05:28:27.153 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-847.g45317f95.el9.n 6/33 2024-04-16T05:28:27.153 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-04-16T05:28:27.153 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-04-16T05:28:27.153 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-04-16T05:28:27.153 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-04-16T05:28:27.154 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-04-16T05:28:27.154 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-04-16T05:28:27.154 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-04-16T05:28:27.154 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-04-16T05:28:27.154 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-04-16T05:28:27.154 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-04-16T05:28:27.154 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-04-16T05:28:27.154 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-04-16T05:28:27.154 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-04-16T05:28:27.154 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-04-16T05:28:27.154 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-04-16T05:28:27.154 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-04-16T05:28:27.154 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-04-16T05:28:27.154 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-04-16T05:28:27.155 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-04-16T05:28:27.155 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-04-16T05:28:27.155 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-04-16T05:28:27.155 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-04-16T05:28:27.155 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-04-16T05:28:27.155 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-04-16T05:28:27.155 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-04-16T05:28:27.155 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-04-16T05:28:27.188 INFO:teuthology.orchestra.run.smithi027.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 436 kB/s | 188 kB 00:00 2024-04-16T05:28:27.199 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:28:27.199 INFO:teuthology.orchestra.run.smithi027.stdout:Total 2.9 MB/s | 3.7 MB 00:01 2024-04-16T05:28:27.200 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-16T05:28:27.228 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-16T05:28:27.229 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-16T05:28:27.375 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-16T05:28:27.375 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-16T05:28:27.791 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-16T05:28:27.923 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-16T05:28:27.977 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-16T05:28:28.083 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-16T05:28:28.147 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-16T05:28:28.215 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-847.g45317f95.el9.no 5/7 2024-04-16T05:28:28.246 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-04-16T05:28:28.246 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:28.246 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-16T05:28:28.246 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:28.246 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:28.246 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:28.246 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T05:28:28.246 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:28.246 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:28.246 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T05:28:28.246 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T05:28:28.247 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T05:28:28.247 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T05:28:28.247 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T05:28:28.247 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T05:28:28.247 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T05:28:28.247 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T05:28:28.247 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T05:28:28.247 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T05:28:28.247 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T05:28:28.247 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T05:28:28.247 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T05:28:28.247 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T05:28:28.247 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T05:28:28.247 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T05:28:28.247 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T05:28:28.247 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T05:28:28.248 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T05:28:28.248 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T05:28:28.248 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T05:28:28.248 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T05:28:28.248 INFO:teuthology.orchestra.run.smithi122.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T05:28:28.248 INFO:teuthology.orchestra.run.smithi122.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T05:28:28.248 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T05:28:28.248 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T05:28:28.248 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T05:28:28.248 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:28.248 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:28:28.663 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-base 2024-04-16T05:28:29.483 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-847.g45317f95.el9.n 6/7 2024-04-16T05:28:29.526 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 7/7 2024-04-16T05:28:29.542 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:00 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:28:29.646 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-base-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T05:28:29.698 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:28:29.699 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-16T05:28:29.700 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:28:29.766 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install cephadm 2024-04-16T05:28:30.613 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:01 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:28:30.769 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 7/7 2024-04-16T05:28:30.769 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-847.g45317f95.el9.n 1/7 2024-04-16T05:28:30.770 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 2/7 2024-04-16T05:28:30.770 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-847.g45317f95.el9.no 3/7 2024-04-16T05:28:30.770 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-16T05:28:30.770 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-16T05:28:30.770 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-16T05:28:30.771 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:28:30.771 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:28:30.771 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-16T05:28:30.771 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:28:30.771 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-16T05:28:30.771 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 82 k 2024-04-16T05:28:30.771 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:30.771 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-16T05:28:30.771 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:28:30.771 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-16T05:28:30.771 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:30.771 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 82 k 2024-04-16T05:28:30.771 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 372 k 2024-04-16T05:28:30.772 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-16T05:28:30.995 INFO:teuthology.orchestra.run.smithi122.stdout:cephadm-17.2.7-847.g45317f95.el9.noarch.rpm 368 kB/s | 82 kB 00:00 2024-04-16T05:28:30.996 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:28:30.997 INFO:teuthology.orchestra.run.smithi122.stdout:Total 363 kB/s | 82 kB 00:00 2024-04-16T05:28:30.998 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-16T05:28:31.014 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-16T05:28:31.014 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-16T05:28:31.021 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-16T05:28:31.022 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-16T05:28:31.118 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-16T05:28:31.118 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:31.118 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-16T05:28:31.118 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T05:28:31.118 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T05:28:31.118 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T05:28:31.118 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-16T05:28:31.119 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-16T05:28:31.119 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T05:28:31.119 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T05:28:31.119 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:31.119 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:28:31.143 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-16T05:28:31.558 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T05:28:31.868 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T05:28:32.000 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T05:28:32.391 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T05:28:32.472 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:19 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:28:32.642 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:28:32.644 INFO:teuthology.orchestra.run.smithi027.stdout:=========================================================================================== 2024-04-16T05:28:32.644 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-04-16T05:28:32.644 INFO:teuthology.orchestra.run.smithi027.stdout:=========================================================================================== 2024-04-16T05:28:32.644 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-16T05:28:32.644 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 7.4 M 2024-04-16T05:28:32.644 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-04-16T05:28:32.644 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T05:28:32.644 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T05:28:32.644 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T05:28:32.644 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T05:28:32.645 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T05:28:32.645 INFO:teuthology.orchestra.run.smithi027.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T05:28:32.645 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T05:28:32.645 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T05:28:32.645 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T05:28:32.645 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T05:28:32.645 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T05:28:32.645 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:32.645 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-16T05:28:32.645 INFO:teuthology.orchestra.run.smithi027.stdout:=========================================================================================== 2024-04-16T05:28:32.645 INFO:teuthology.orchestra.run.smithi027.stdout:Install 12 Packages 2024-04-16T05:28:32.645 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:32.646 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 37 M 2024-04-16T05:28:32.646 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 209 M 2024-04-16T05:28:32.646 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-16T05:28:32.733 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/1 2024-04-16T05:28:32.734 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:32.734 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-16T05:28:32.734 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T05:28:32.734 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:32.734 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:28:32.972 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T05:28:33.297 INFO:teuthology.orchestra.run.smithi027.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 141 kB/s | 32 kB 00:00 2024-04-16T05:28:33.340 INFO:teuthology.orchestra.run.smithi027.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 457 kB/s | 17 kB 00:00 2024-04-16T05:28:33.439 INFO:teuthology.orchestra.run.smithi027.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.2 MB/s | 3.0 MB 00:00 2024-04-16T05:28:33.465 INFO:teuthology.orchestra.run.smithi027.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 319 kB/s | 34 kB 00:00 2024-04-16T05:28:33.657 INFO:teuthology.orchestra.run.smithi027.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2024-04-16T05:28:33.749 INFO:teuthology.orchestra.run.smithi027.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-04-16T05:28:33.782 INFO:teuthology.orchestra.run.smithi027.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.8 MB/s | 217 kB 00:00 2024-04-16T05:28:33.844 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:04 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:28:34.000 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:28:34.001 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:28:34.001 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-04-16T05:28:34.001 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:28:34.001 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-16T05:28:34.001 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-847.g45317f95.el9 ceph 137 k 2024-04-16T05:28:34.001 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:34.001 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-16T05:28:34.001 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:28:34.001 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-16T05:28:34.001 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:34.002 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 137 k 2024-04-16T05:28:34.002 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 379 k 2024-04-16T05:28:34.002 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-16T05:28:34.132 INFO:teuthology.orchestra.run.smithi027.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-04-16T05:28:34.358 INFO:teuthology.orchestra.run.smithi027.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-04-16T05:28:34.367 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-immutable-object-cache-17.2.7-847.g45317f9 377 kB/s | 137 kB 00:00 2024-04-16T05:28:34.369 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:28:34.370 INFO:teuthology.orchestra.run.smithi122.stdout:Total 373 kB/s | 137 kB 00:00 2024-04-16T05:28:34.370 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-16T05:28:34.395 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-16T05:28:34.395 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-16T05:28:34.400 INFO:teuthology.orchestra.run.smithi027.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 742 kB/s | 195 kB 00:00 2024-04-16T05:28:34.492 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-16T05:28:34.492 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-16T05:28:34.666 INFO:teuthology.orchestra.run.smithi027.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 800 kB/s | 245 kB 00:00 2024-04-16T05:28:34.693 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-16T05:28:34.777 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-847.g45317f95.e 1/1 2024-04-16T05:28:34.853 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-847.g45317f95.e 1/1 2024-04-16T05:28:34.936 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T05:28:34.937 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T05:28:34.937 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:35.092 INFO:teuthology.orchestra.run.smithi027.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-8 3.7 MB/s | 7.4 MB 00:02 2024-04-16T05:28:35.101 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:28:35.101 INFO:teuthology.orchestra.run.smithi027.stdout:Total 15 MB/s | 37 MB 00:02 2024-04-16T05:28:35.222 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-16T05:28:35.292 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-16T05:28:35.292 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-16T05:28:35.828 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-16T05:28:35.829 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-16T05:28:36.208 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-847.g45317f95.e 1/1 2024-04-16T05:28:36.208 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:36.208 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-16T05:28:36.208 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:28:36.208 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:36.208 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:28:36.481 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr 2024-04-16T05:28:36.824 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-16T05:28:36.922 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T05:28:36.974 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T05:28:37.131 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T05:28:37.169 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T05:28:37.346 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:08 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:28:37.441 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T05:28:37.449 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-mgr-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T05:28:37.475 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T05:28:37.502 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:28:37.503 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-16T05:28:37.503 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:28:37.578 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T05:28:37.605 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T05:28:38.424 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:09 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:28:38.527 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T05:28:38.582 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:28:38.583 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:28:38.583 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-16T05:28:38.583 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:28:38.584 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-16T05:28:38.584 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 3.4 M 2024-04-16T05:28:38.584 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-16T05:28:38.584 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 24 k 2024-04-16T05:28:38.584 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 15 k 2024-04-16T05:28:38.584 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-04-16T05:28:38.584 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T05:28:38.584 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T05:28:38.584 INFO:teuthology.orchestra.run.smithi122.stdout:Installing weak dependencies: 2024-04-16T05:28:38.584 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-04-16T05:28:38.584 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:38.584 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-16T05:28:38.584 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:28:38.584 INFO:teuthology.orchestra.run.smithi122.stdout:Install 7 Packages 2024-04-16T05:28:38.585 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:38.585 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 3.7 M 2024-04-16T05:28:38.585 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 80 M 2024-04-16T05:28:38.585 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-16T05:28:38.692 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T05:28:39.071 INFO:teuthology.orchestra.run.smithi122.stdout:(1/7): ceph-grafana-dashboards-17.2.7-847.g4531 73 kB/s | 24 kB 00:00 2024-04-16T05:28:39.104 INFO:teuthology.orchestra.run.smithi122.stdout:(2/7): ceph-prometheus-alerts-17.2.7-847.g45317 40 kB/s | 15 kB 00:00 2024-04-16T05:28:39.413 INFO:teuthology.orchestra.run.smithi122.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 27 kB/s | 9.0 kB 00:00 2024-04-16T05:28:39.555 INFO:teuthology.orchestra.run.smithi122.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 91 kB/s | 41 kB 00:00 2024-04-16T05:28:39.580 INFO:teuthology.orchestra.run.smithi122.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 185 kB/s | 31 kB 00:00 2024-04-16T05:28:39.697 INFO:teuthology.orchestra.run.smithi122.stdout:(6/7): ceph-mgr-dashboard-17.2.7-847.g45317f95. 3.5 MB/s | 3.4 MB 00:00 2024-04-16T05:28:39.822 INFO:teuthology.orchestra.run.smithi122.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 709 kB/s | 188 kB 00:00 2024-04-16T05:28:39.827 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:28:39.828 INFO:teuthology.orchestra.run.smithi122.stdout:Total 2.9 MB/s | 3.7 MB 00:01 2024-04-16T05:28:39.828 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-16T05:28:39.843 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-16T05:28:39.843 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-16T05:28:39.959 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-16T05:28:39.960 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-16T05:28:40.341 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T05:28:40.359 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-16T05:28:40.424 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-04-16T05:28:40.469 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-04-16T05:28:40.554 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-04-16T05:28:40.605 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-04-16T05:28:40.647 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-847.g45317f95.el9.no 5/7 2024-04-16T05:28:41.712 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T05:28:41.756 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f 12/12 2024-04-16T05:28:41.852 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-847.g45317f95.el9.n 6/7 2024-04-16T05:28:41.902 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 7/7 2024-04-16T05:28:43.105 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 7/7 2024-04-16T05:28:43.106 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-847.g45317f95.el9.n 1/7 2024-04-16T05:28:43.106 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 2/7 2024-04-16T05:28:43.106 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-847.g45317f95.el9.no 3/7 2024-04-16T05:28:43.106 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-04-16T05:28:43.106 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-04-16T05:28:43.106 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-04-16T05:28:43.462 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-04-16T05:28:43.462 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:43.462 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-16T05:28:43.462 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T05:28:43.462 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T05:28:43.462 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T05:28:43.463 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-04-16T05:28:43.463 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-04-16T05:28:43.463 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T05:28:43.463 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T05:28:43.463 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:43.463 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:28:43.759 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T05:28:44.323 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f 12/12 2024-04-16T05:28:44.323 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f 1/12 2024-04-16T05:28:44.324 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T05:28:44.324 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T05:28:44.324 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T05:28:44.324 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T05:28:44.324 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T05:28:44.324 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T05:28:44.324 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T05:28:44.324 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T05:28:44.324 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T05:28:44.324 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T05:28:44.593 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:15 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:28:44.756 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:28:44.758 INFO:teuthology.orchestra.run.smithi122.stdout:=========================================================================================== 2024-04-16T05:28:44.758 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-16T05:28:44.758 INFO:teuthology.orchestra.run.smithi122.stdout:=========================================================================================== 2024-04-16T05:28:44.758 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-16T05:28:44.758 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 7.4 M 2024-04-16T05:28:44.758 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-16T05:28:44.758 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T05:28:44.758 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T05:28:44.758 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T05:28:44.759 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T05:28:44.759 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T05:28:44.759 INFO:teuthology.orchestra.run.smithi122.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T05:28:44.759 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T05:28:44.759 INFO:teuthology.orchestra.run.smithi122.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T05:28:44.759 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T05:28:44.759 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T05:28:44.759 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T05:28:44.759 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:44.759 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-16T05:28:44.759 INFO:teuthology.orchestra.run.smithi122.stdout:=========================================================================================== 2024-04-16T05:28:44.759 INFO:teuthology.orchestra.run.smithi122.stdout:Install 12 Packages 2024-04-16T05:28:44.760 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:44.760 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 37 M 2024-04-16T05:28:44.760 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 209 M 2024-04-16T05:28:44.760 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-16T05:28:44.787 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T05:28:44.787 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:44.787 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-16T05:28:44.787 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T05:28:44.787 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T05:28:44.787 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T05:28:44.787 INFO:teuthology.orchestra.run.smithi027.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T05:28:44.787 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T05:28:44.788 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T05:28:44.788 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T05:28:44.788 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T05:28:44.788 INFO:teuthology.orchestra.run.smithi027.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T05:28:44.788 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T05:28:44.788 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T05:28:44.788 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T05:28:44.788 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:44.789 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:28:45.149 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-rook 2024-04-16T05:28:45.432 INFO:teuthology.orchestra.run.smithi122.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 144 kB/s | 32 kB 00:00 2024-04-16T05:28:45.473 INFO:teuthology.orchestra.run.smithi122.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 450 kB/s | 17 kB 00:00 2024-04-16T05:28:45.565 INFO:teuthology.orchestra.run.smithi122.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.5 MB/s | 3.0 MB 00:00 2024-04-16T05:28:45.590 INFO:teuthology.orchestra.run.smithi122.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 335 kB/s | 34 kB 00:00 2024-04-16T05:28:45.766 INFO:teuthology.orchestra.run.smithi122.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 24 MB/s | 4.7 MB 00:00 2024-04-16T05:28:45.875 INFO:teuthology.orchestra.run.smithi122.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-04-16T05:28:45.908 INFO:teuthology.orchestra.run.smithi122.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.6 MB/s | 217 kB 00:00 2024-04-16T05:28:46.074 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:33 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:28:46.226 INFO:teuthology.orchestra.run.smithi122.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-04-16T05:28:46.241 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:28:46.243 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-04-16T05:28:46.243 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-04-16T05:28:46.243 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-04-16T05:28:46.243 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-16T05:28:46.243 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 47 k 2024-04-16T05:28:46.243 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-04-16T05:28:46.243 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T05:28:46.243 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T05:28:46.243 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T05:28:46.243 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T05:28:46.243 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T05:28:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T05:28:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T05:28:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T05:28:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T05:28:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T05:28:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T05:28:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T05:28:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:46.244 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-16T05:28:46.244 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-04-16T05:28:46.244 INFO:teuthology.orchestra.run.smithi027.stdout:Install 13 Packages 2024-04-16T05:28:46.244 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:46.245 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 2.0 M 2024-04-16T05:28:46.245 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 26 M 2024-04-16T05:28:46.245 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-16T05:28:46.392 INFO:teuthology.orchestra.run.smithi122.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-04-16T05:28:46.426 INFO:teuthology.orchestra.run.smithi122.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 982 kB/s | 195 kB 00:00 2024-04-16T05:28:46.643 INFO:teuthology.orchestra.run.smithi122.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 985 kB/s | 245 kB 00:00 2024-04-16T05:28:46.657 INFO:teuthology.orchestra.run.smithi027.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 117 kB/s | 24 kB 00:00 2024-04-16T05:28:46.682 INFO:teuthology.orchestra.run.smithi027.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 78 kB/s | 18 kB 00:00 2024-04-16T05:28:46.756 INFO:teuthology.orchestra.run.smithi027.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.8 MB/s | 174 kB 00:00 2024-04-16T05:28:46.783 INFO:teuthology.orchestra.run.smithi027.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 132 kB 00:00 2024-04-16T05:28:46.825 INFO:teuthology.orchestra.run.smithi027.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.1 MB/s | 211 kB 00:00 2024-04-16T05:28:46.850 INFO:teuthology.orchestra.run.smithi027.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 668 kB/s | 43 kB 00:00 2024-04-16T05:28:46.875 INFO:teuthology.orchestra.run.smithi027.stdout:(7/13): ceph-mgr-rook-17.2.7-847.g45317f95.el9. 111 kB/s | 47 kB 00:00 2024-04-16T05:28:47.185 INFO:teuthology.orchestra.run.smithi122.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-8 3.7 MB/s | 7.4 MB 00:01 2024-04-16T05:28:47.190 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:28:47.190 INFO:teuthology.orchestra.run.smithi122.stdout:Total 15 MB/s | 37 MB 00:02 2024-04-16T05:28:47.201 INFO:teuthology.orchestra.run.smithi027.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 43 kB/s | 15 kB 00:00 2024-04-16T05:28:47.226 INFO:teuthology.orchestra.run.smithi027.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 81 kB/s | 32 kB 00:00 2024-04-16T05:28:47.310 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-16T05:28:47.368 INFO:teuthology.orchestra.run.smithi027.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 419 kB/s | 59 kB 00:00 2024-04-16T05:28:47.372 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-16T05:28:47.372 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-16T05:28:47.402 INFO:teuthology.orchestra.run.smithi027.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 386 kB/s | 203 kB 00:00 2024-04-16T05:28:47.477 INFO:teuthology.orchestra.run.smithi027.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 834 kB/s | 90 kB 00:00 2024-04-16T05:28:47.586 INFO:teuthology.orchestra.run.smithi027.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.7 MB/s | 1.0 MB 00:00 2024-04-16T05:28:47.596 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:28:47.597 INFO:teuthology.orchestra.run.smithi027.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-04-16T05:28:47.598 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-16T05:28:47.641 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-16T05:28:47.641 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-16T05:28:47.809 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-16T05:28:47.810 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-16T05:28:47.900 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-16T05:28:47.900 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-16T05:28:48.511 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-16T05:28:48.740 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T05:28:48.746 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-16T05:28:48.830 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T05:28:48.865 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T05:28:48.968 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T05:28:49.008 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T05:28:49.051 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T05:28:49.052 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T05:28:49.093 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T05:28:49.144 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T05:28:49.281 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T05:28:49.355 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T05:28:49.392 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T05:28:49.429 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T05:28:49.514 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T05:28:49.641 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T05:28:50.362 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T05:28:50.495 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T05:28:50.639 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T05:28:50.697 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T05:28:50.730 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T05:28:50.784 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T05:28:50.806 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 13/13 2024-04-16T05:28:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 13/13 2024-04-16T05:28:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 1/13 2024-04-16T05:28:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T05:28:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T05:28:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T05:28:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T05:28:51.907 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T05:28:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T05:28:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T05:28:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T05:28:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T05:28:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T05:28:51.908 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T05:28:52.030 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T05:28:52.371 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T05:28:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:52.372 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-16T05:28:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T05:28:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T05:28:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T05:28:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T05:28:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T05:28:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T05:28:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T05:28:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T05:28:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T05:28:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T05:28:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T05:28:52.372 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T05:28:52.373 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T05:28:52.373 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:52.373 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:28:52.681 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T05:28:53.188 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T05:28:53.224 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f 12/12 2024-04-16T05:28:53.582 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:40 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:28:53.753 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:28:53.754 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-04-16T05:28:53.754 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-04-16T05:28:53.754 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-04-16T05:28:53.754 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-16T05:28:53.754 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 123 k 2024-04-16T05:28:53.754 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-04-16T05:28:53.754 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T05:28:53.754 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T05:28:53.754 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T05:28:53.754 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T05:28:53.755 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T05:28:53.755 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:53.755 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-16T05:28:53.755 INFO:teuthology.orchestra.run.smithi027.stdout:======================================================================================= 2024-04-16T05:28:53.755 INFO:teuthology.orchestra.run.smithi027.stdout:Install 6 Packages 2024-04-16T05:28:53.755 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:53.755 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 6.8 M 2024-04-16T05:28:53.756 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 33 M 2024-04-16T05:28:53.756 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-16T05:28:54.671 INFO:teuthology.orchestra.run.smithi027.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2024-04-16T05:28:54.780 INFO:teuthology.orchestra.run.smithi027.stdout:(2/6): ceph-mgr-cephadm-17.2.7-847.g45317f95.el 264 kB/s | 123 kB 00:00 2024-04-16T05:28:54.880 INFO:teuthology.orchestra.run.smithi027.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 583 kB/s | 58 kB 00:00 2024-04-16T05:28:54.997 INFO:teuthology.orchestra.run.smithi027.stdout:(4/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 363 kB/s | 248 kB 00:00 2024-04-16T05:28:55.155 INFO:teuthology.orchestra.run.smithi027.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-04-16T05:28:55.355 INFO:teuthology.orchestra.run.smithi027.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 107 kB/s | 51 kB 00:00 2024-04-16T05:28:55.374 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:28:55.374 INFO:teuthology.orchestra.run.smithi027.stdout:Total 4.2 MB/s | 6.8 MB 00:01 2024-04-16T05:28:55.402 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-16T05:28:55.440 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-16T05:28:55.440 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-16T05:28:55.593 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-16T05:28:55.594 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-16T05:28:55.770 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f 12/12 2024-04-16T05:28:55.770 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f 1/12 2024-04-16T05:28:55.770 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T05:28:55.771 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T05:28:55.771 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T05:28:55.771 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T05:28:55.771 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T05:28:55.771 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T05:28:55.771 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T05:28:55.771 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T05:28:55.771 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T05:28:55.771 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T05:28:55.995 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-16T05:28:56.194 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T05:28:56.194 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:56.194 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-16T05:28:56.194 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T05:28:56.194 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T05:28:56.195 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T05:28:56.195 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T05:28:56.195 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T05:28:56.195 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T05:28:56.195 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T05:28:56.195 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T05:28:56.195 INFO:teuthology.orchestra.run.smithi122.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T05:28:56.195 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T05:28:56.195 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T05:28:56.195 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T05:28:56.195 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:56.195 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:28:56.238 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T05:28:56.298 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T05:28:56.605 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-rook 2024-04-16T05:28:57.034 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T05:28:57.136 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T05:28:57.208 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T05:28:57.244 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 6/6 2024-04-16T05:28:57.502 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:28 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:28:57.663 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:28:57.664 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-04-16T05:28:57.664 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-16T05:28:57.664 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-04-16T05:28:57.664 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-16T05:28:57.664 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 47 k 2024-04-16T05:28:57.665 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-16T05:28:57.665 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T05:28:57.665 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T05:28:57.665 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T05:28:57.665 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T05:28:57.665 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T05:28:57.665 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T05:28:57.665 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T05:28:57.665 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T05:28:57.665 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T05:28:57.665 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T05:28:57.665 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T05:28:57.665 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T05:28:57.665 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:57.666 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-16T05:28:57.666 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-04-16T05:28:57.666 INFO:teuthology.orchestra.run.smithi122.stdout:Install 13 Packages 2024-04-16T05:28:57.666 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:28:57.666 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 2.0 M 2024-04-16T05:28:57.666 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 26 M 2024-04-16T05:28:57.666 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-16T05:28:58.032 INFO:teuthology.orchestra.run.smithi122.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 136 kB/s | 24 kB 00:00 2024-04-16T05:28:58.057 INFO:teuthology.orchestra.run.smithi122.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 89 kB/s | 18 kB 00:00 2024-04-16T05:28:58.116 INFO:teuthology.orchestra.run.smithi122.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-04-16T05:28:58.141 INFO:teuthology.orchestra.run.smithi122.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2024-04-16T05:28:58.175 INFO:teuthology.orchestra.run.smithi122.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.6 MB/s | 211 kB 00:00 2024-04-16T05:28:58.200 INFO:teuthology.orchestra.run.smithi122.stdout:(6/13): ceph-mgr-rook-17.2.7-847.g45317f95.el9. 137 kB/s | 47 kB 00:00 2024-04-16T05:28:58.226 INFO:teuthology.orchestra.run.smithi122.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 528 kB/s | 43 kB 00:00 2024-04-16T05:28:58.240 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 6/6 2024-04-16T05:28:58.240 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/6 2024-04-16T05:28:58.240 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T05:28:58.240 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T05:28:58.240 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T05:28:58.240 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T05:28:58.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T05:28:58.667 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:58.667 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-16T05:28:58.667 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T05:28:58.668 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T05:28:58.668 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T05:28:58.668 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T05:28:58.668 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T05:28:58.668 INFO:teuthology.orchestra.run.smithi027.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T05:28:58.668 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:28:58.668 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:28:58.964 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-fuse 2024-04-16T05:28:59.888 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:46 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:29:00.057 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:29:00.058 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:29:00.058 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-04-16T05:29:00.058 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:29:00.058 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-16T05:29:00.059 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse x86_64 2:17.2.7-847.g45317f95.el9 ceph 813 k 2024-04-16T05:29:00.059 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-04-16T05:29:00.059 INFO:teuthology.orchestra.run.smithi027.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T05:29:00.059 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:00.059 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-16T05:29:00.059 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:29:00.059 INFO:teuthology.orchestra.run.smithi027.stdout:Install 2 Packages 2024-04-16T05:29:00.059 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:00.060 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 891 k 2024-04-16T05:29:00.060 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 2.6 M 2024-04-16T05:29:00.060 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-16T05:29:00.200 INFO:teuthology.orchestra.run.smithi027.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 565 kB/s | 79 kB 00:00 2024-04-16T05:29:00.253 INFO:teuthology.orchestra.run.smithi122.stdout:[MIRROR] python3-cachetools-4.2.4-1.el9.noarch.rpm: Curl error (7): Couldn't connect to server for https://veronanetworks.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/python3-cachetools-4.2.4-1.el9.noarch.rpm [Failed to connect to veronanetworks.mm.fcix.net port 443: Network is unreachable] 2024-04-16T05:29:00.254 INFO:teuthology.orchestra.run.smithi122.stdout:[MIRROR] python3-certifi-2021.10.8-2.el9.noarch.rpm: Curl error (7): Couldn't connect to server for https://veronanetworks.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/python3-certifi-2021.10.8-2.el9.noarch.rpm [Failed to connect to veronanetworks.mm.fcix.net port 443: Network is unreachable] 2024-04-16T05:29:00.254 INFO:teuthology.orchestra.run.smithi122.stdout:[MIRROR] python3-google-auth-2.28.2-1.el9.noarch.rpm: Curl error (7): Couldn't connect to server for https://veronanetworks.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/python3-google-auth-2.28.2-1.el9.noarch.rpm [Failed to connect to veronanetworks.mm.fcix.net port 443: Network is unreachable] 2024-04-16T05:29:00.352 INFO:teuthology.orchestra.run.smithi122.stdout:[MIRROR] python3-cachetools-4.2.4-1.el9.noarch.rpm: Curl error (7): Couldn't connect to server for http://veronanetworks.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/python3-cachetools-4.2.4-1.el9.noarch.rpm [Failed to connect to veronanetworks.mm.fcix.net port 80: Network is unreachable] 2024-04-16T05:29:00.353 INFO:teuthology.orchestra.run.smithi122.stdout:[MIRROR] python3-certifi-2021.10.8-2.el9.noarch.rpm: Curl error (7): Couldn't connect to server for http://veronanetworks.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/python3-certifi-2021.10.8-2.el9.noarch.rpm [Failed to connect to veronanetworks.mm.fcix.net port 80: Network is unreachable] 2024-04-16T05:29:00.353 INFO:teuthology.orchestra.run.smithi122.stdout:[MIRROR] python3-google-auth-2.28.2-1.el9.noarch.rpm: Curl error (7): Couldn't connect to server for http://veronanetworks.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/python3-google-auth-2.28.2-1.el9.noarch.rpm [Failed to connect to veronanetworks.mm.fcix.net port 80: Network is unreachable] 2024-04-16T05:29:00.583 INFO:teuthology.orchestra.run.smithi027.stdout:(2/2): ceph-fuse-17.2.7-847.g45317f95.el9.x86_6 1.5 MB/s | 813 kB 00:00 2024-04-16T05:29:00.587 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:29:00.588 INFO:teuthology.orchestra.run.smithi027.stdout:Total 1.6 MB/s | 891 kB 00:00 2024-04-16T05:29:00.588 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-16T05:29:00.592 INFO:teuthology.orchestra.run.smithi122.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 6.3 kB/s | 15 kB 00:02 2024-04-16T05:29:00.617 INFO:teuthology.orchestra.run.smithi122.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 13 kB/s | 32 kB 00:02 2024-04-16T05:29:00.660 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-16T05:29:00.660 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-16T05:29:00.760 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-16T05:29:00.760 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-16T05:29:00.843 INFO:teuthology.orchestra.run.smithi122.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 77 kB/s | 203 kB 00:02 2024-04-16T05:29:01.036 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-16T05:29:01.199 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T05:29:01.291 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 2/2 2024-04-16T05:29:02.282 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 2/2 2024-04-16T05:29:02.282 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/2 2024-04-16T05:29:02.606 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T05:29:02.606 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:02.606 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-16T05:29:02.606 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T05:29:02.606 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:02.606 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:29:02.939 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librados-devel 2024-04-16T05:29:03.861 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:50 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:29:04.029 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:29:04.029 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:29:04.029 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-04-16T05:29:04.030 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:29:04.030 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-16T05:29:04.030 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel x86_64 2:17.2.7-847.g45317f95.el9 ceph 126 k 2024-04-16T05:29:04.030 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:04.030 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-16T05:29:04.030 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:29:04.030 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-04-16T05:29:04.030 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:04.031 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 126 k 2024-04-16T05:29:04.031 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 448 k 2024-04-16T05:29:04.031 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-16T05:29:04.364 INFO:teuthology.orchestra.run.smithi027.stdout:librados-devel-17.2.7-847.g45317f95.el9.x86_64. 380 kB/s | 126 kB 00:00 2024-04-16T05:29:04.366 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:29:04.367 INFO:teuthology.orchestra.run.smithi027.stdout:Total 375 kB/s | 126 kB 00:00 2024-04-16T05:29:04.368 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-16T05:29:04.387 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-16T05:29:04.388 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-16T05:29:04.511 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-16T05:29:04.512 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-16T05:29:04.721 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-16T05:29:04.846 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:05.474 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:05.788 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:05.788 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:05.788 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-16T05:29:05.788 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:29:05.789 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:05.789 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:29:06.088 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install libcephfs2 2024-04-16T05:29:07.027 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:54 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:29:07.139 INFO:teuthology.orchestra.run.smithi027.stdout:Package libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T05:29:07.195 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:29:07.197 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-16T05:29:07.197 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:29:07.263 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install libcephfs-devel 2024-04-16T05:29:08.169 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:55 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:29:08.336 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:29:08.336 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:29:08.337 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-04-16T05:29:08.337 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:29:08.337 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-16T05:29:08.337 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel x86_64 2:17.2.7-847.g45317f95.el9 ceph 25 k 2024-04-16T05:29:08.337 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:08.337 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-16T05:29:08.337 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:29:08.337 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-04-16T05:29:08.337 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:08.338 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 25 k 2024-04-16T05:29:08.338 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 110 k 2024-04-16T05:29:08.338 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-16T05:29:08.501 INFO:teuthology.orchestra.run.smithi027.stdout:libcephfs-devel-17.2.7-847.g45317f95.el9.x86_64 154 kB/s | 25 kB 00:00 2024-04-16T05:29:08.502 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:29:08.502 INFO:teuthology.orchestra.run.smithi027.stdout:Total 151 kB/s | 25 kB 00:00 2024-04-16T05:29:08.503 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-16T05:29:08.512 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-16T05:29:08.512 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-16T05:29:08.517 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-16T05:29:08.518 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-16T05:29:08.665 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-16T05:29:08.838 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:09.312 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:09.617 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:09.617 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:09.617 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-16T05:29:09.617 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:29:09.617 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:09.617 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:29:09.833 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librados2 2024-04-16T05:29:10.750 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:57 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:29:10.862 INFO:teuthology.orchestra.run.smithi027.stdout:Package librados2-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T05:29:10.921 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:29:10.922 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-16T05:29:10.922 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:29:10.989 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librbd1 2024-04-16T05:29:11.908 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:58 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:29:12.022 INFO:teuthology.orchestra.run.smithi027.stdout:Package librbd1-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T05:29:12.082 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:29:12.084 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-16T05:29:12.084 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:29:12.152 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rados 2024-04-16T05:29:13.077 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:00 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:29:13.185 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T05:29:13.242 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:29:13.244 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-16T05:29:13.244 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:29:13.314 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rgw 2024-04-16T05:29:14.220 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:01 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:29:14.345 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T05:29:14.403 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:29:14.405 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-16T05:29:14.405 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:29:14.478 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-cephfs 2024-04-16T05:29:15.360 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:02 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:29:15.469 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T05:29:15.526 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:29:15.528 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-16T05:29:15.528 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:29:15.598 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rbd 2024-04-16T05:29:16.505 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:03 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:29:16.613 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T05:29:16.670 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:29:16.672 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-04-16T05:29:16.672 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:29:16.748 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-fuse 2024-04-16T05:29:17.626 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:04 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:29:17.797 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:29:17.798 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:29:17.798 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-04-16T05:29:17.798 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:29:17.798 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-16T05:29:17.798 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse x86_64 2:17.2.7-847.g45317f95.el9 ceph 82 k 2024-04-16T05:29:17.798 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:17.798 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-16T05:29:17.798 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:29:17.798 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-04-16T05:29:17.798 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:17.798 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 82 k 2024-04-16T05:29:17.798 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 222 k 2024-04-16T05:29:17.799 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-16T05:29:18.021 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-fuse-17.2.7-847.g45317f95.el9.x86_64.rpm 368 kB/s | 82 kB 00:00 2024-04-16T05:29:18.023 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:29:18.023 INFO:teuthology.orchestra.run.smithi027.stdout:Total 363 kB/s | 82 kB 00:00 2024-04-16T05:29:18.024 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-16T05:29:18.049 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-16T05:29:18.049 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-16T05:29:18.152 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-16T05:29:18.152 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-16T05:29:18.342 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-16T05:29:18.494 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:19.057 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:19.323 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:19.323 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:19.323 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-16T05:29:19.324 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:29:19.324 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:19.324 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:29:19.569 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-mirror 2024-04-16T05:29:19.966 INFO:teuthology.orchestra.run.smithi122.stdout:[MIRROR] python3-kubernetes-26.1.0-2.el9.noarch.rpm: Curl error (7): Couldn't connect to server for http://veronanetworks.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/python3-kubernetes-26.1.0-2.el9.noarch.rpm [Failed to connect to veronanetworks.mm.fcix.net port 80: Network is unreachable] 2024-04-16T05:29:19.967 INFO:teuthology.orchestra.run.smithi122.stdout:[MIRROR] python3-rsa-4.9-2.el9.noarch.rpm: Curl error (7): Couldn't connect to server for http://veronanetworks.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/python3-rsa-4.9-2.el9.noarch.rpm [Failed to connect to veronanetworks.mm.fcix.net port 80: Network is unreachable] 2024-04-16T05:29:19.994 INFO:teuthology.orchestra.run.smithi122.stdout:[MIRROR] python3-kubernetes-26.1.0-2.el9.noarch.rpm: Curl error (7): Couldn't connect to server for https://veronanetworks.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/python3-kubernetes-26.1.0-2.el9.noarch.rpm [Failed to connect to veronanetworks.mm.fcix.net port 443: Network is unreachable] 2024-04-16T05:29:19.994 INFO:teuthology.orchestra.run.smithi122.stdout:[MIRROR] python3-rsa-4.9-2.el9.noarch.rpm: Curl error (7): Couldn't connect to server for https://veronanetworks.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/python3-rsa-4.9-2.el9.noarch.rpm [Failed to connect to veronanetworks.mm.fcix.net port 443: Network is unreachable] 2024-04-16T05:29:20.324 INFO:teuthology.orchestra.run.smithi122.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 3.0 kB/s | 59 kB 00:19 2024-04-16T05:29:20.477 INFO:teuthology.orchestra.run.smithi122.stdout:[MIRROR] python3-websocket-client-1.2.3-2.el9.noarch.rpm: Curl error (7): Couldn't connect to server for http://veronanetworks.mm.fcix.net/epel/9/Everything/x86_64/Packages/p/python3-websocket-client-1.2.3-2.el9.noarch.rpm [Failed to connect to veronanetworks.mm.fcix.net port 80: Network is unreachable] 2024-04-16T05:29:20.485 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:07 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:29:20.650 INFO:teuthology.orchestra.run.smithi122.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 52 kB/s | 1.0 MB 00:20 2024-04-16T05:29:20.652 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:29:20.653 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:29:20.653 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-04-16T05:29:20.653 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:29:20.653 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-16T05:29:20.653 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror x86_64 2:17.2.7-847.g45317f95.el9 ceph 2.9 M 2024-04-16T05:29:20.653 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:20.653 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-16T05:29:20.653 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:29:20.654 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-04-16T05:29:20.654 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:20.654 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 2.9 M 2024-04-16T05:29:20.654 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 12 M 2024-04-16T05:29:20.654 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-16T05:29:20.675 INFO:teuthology.orchestra.run.smithi122.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 4.5 kB/s | 90 kB 00:19 2024-04-16T05:29:20.679 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:29:20.680 INFO:teuthology.orchestra.run.smithi122.stdout:Total 91 kB/s | 2.0 MB 00:23 2024-04-16T05:29:20.681 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-16T05:29:20.722 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-16T05:29:20.723 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-16T05:29:20.880 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-16T05:29:20.880 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-16T05:29:21.149 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-mirror-17.2.7-847.g45317f95.el9.x86_64.rpm 5.9 MB/s | 2.9 MB 00:00 2024-04-16T05:29:21.151 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:29:21.152 INFO:teuthology.orchestra.run.smithi027.stdout:Total 5.9 MB/s | 2.9 MB 00:00 2024-04-16T05:29:21.152 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-16T05:29:21.185 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-16T05:29:21.185 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-16T05:29:21.316 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-16T05:29:21.317 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-16T05:29:21.541 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-16T05:29:21.767 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T05:29:21.835 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T05:29:21.841 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-16T05:29:21.899 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:21.902 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T05:29:21.947 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T05:29:21.988 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:21.988 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T05:29:21.988 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T05:29:21.988 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T05:29:21.988 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T05:29:21.988 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:21.993 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T05:29:22.121 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T05:29:22.261 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T05:29:22.336 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T05:29:23.262 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T05:29:23.328 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T05:29:23.360 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T05:29:23.415 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T05:29:23.437 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 13/13 2024-04-16T05:29:23.559 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:23.559 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:23.559 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-16T05:29:23.559 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:29:23.559 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:23.559 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:29:23.833 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-nbd 2024-04-16T05:29:24.326 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 13/13 2024-04-16T05:29:24.326 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 1/13 2024-04-16T05:29:24.326 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T05:29:24.326 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T05:29:24.327 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T05:29:24.327 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T05:29:24.327 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T05:29:24.327 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T05:29:24.327 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T05:29:24.327 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T05:29:24.327 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T05:29:24.327 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T05:29:24.327 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T05:29:24.750 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:02:11 ago on Tue 16 Apr 2024 05:27:13 AM UTC. 2024-04-16T05:29:24.788 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T05:29:24.789 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:24.789 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-16T05:29:24.789 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T05:29:24.789 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T05:29:24.789 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T05:29:24.789 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T05:29:24.789 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T05:29:24.789 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T05:29:24.789 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T05:29:24.789 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T05:29:24.789 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T05:29:24.789 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T05:29:24.790 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T05:29:24.790 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T05:29:24.790 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T05:29:24.790 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:24.790 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:29:24.922 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-04-16T05:29:24.923 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:29:24.923 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-04-16T05:29:24.923 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:29:24.923 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-04-16T05:29:24.923 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd x86_64 2:17.2.7-847.g45317f95.el9 ceph 165 k 2024-04-16T05:29:24.923 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:24.923 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-04-16T05:29:24.924 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-04-16T05:29:24.924 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-04-16T05:29:24.924 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:24.924 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 165 k 2024-04-16T05:29:24.924 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 477 k 2024-04-16T05:29:24.924 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-04-16T05:29:25.081 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T05:29:25.302 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-nbd-17.2.7-847.g45317f95.el9.x86_64.rpm 437 kB/s | 165 kB 00:00 2024-04-16T05:29:25.304 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:29:25.305 INFO:teuthology.orchestra.run.smithi027.stdout:Total 433 kB/s | 165 kB 00:00 2024-04-16T05:29:25.305 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-04-16T05:29:25.333 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-04-16T05:29:25.333 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-04-16T05:29:25.439 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-04-16T05:29:25.439 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-04-16T05:29:25.670 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-04-16T05:29:25.814 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:25.953 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:56 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:29:26.115 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:29:26.116 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-04-16T05:29:26.116 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-16T05:29:26.116 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-04-16T05:29:26.116 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-16T05:29:26.116 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm noarch 2:17.2.7-847.g45317f95.el9 ceph-noarch 123 k 2024-04-16T05:29:26.116 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-16T05:29:26.116 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T05:29:26.116 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T05:29:26.116 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T05:29:26.117 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T05:29:26.117 INFO:teuthology.orchestra.run.smithi122.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T05:29:26.117 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:26.117 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-16T05:29:26.117 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================= 2024-04-16T05:29:26.117 INFO:teuthology.orchestra.run.smithi122.stdout:Install 6 Packages 2024-04-16T05:29:26.117 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:26.118 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 6.8 M 2024-04-16T05:29:26.118 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 33 M 2024-04-16T05:29:26.118 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-16T05:29:26.450 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:26.735 INFO:teuthology.orchestra.run.smithi122.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 817 kB/s | 248 kB 00:00 2024-04-16T05:29:26.786 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:26.787 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:26.787 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-04-16T05:29:26.787 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:29:26.787 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:29:26.787 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-04-16T05:29:26.827 INFO:teuthology.orchestra.run.smithi122.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2024-04-16T05:29:26.919 INFO:teuthology.orchestra.run.smithi122.stdout:(3/6): ceph-mgr-cephadm-17.2.7-847.g45317f95.el 253 kB/s | 123 kB 00:00 2024-04-16T05:29:26.952 INFO:teuthology.orchestra.run.smithi122.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 464 kB/s | 58 kB 00:00 2024-04-16T05:29:27.053 DEBUG:teuthology.parallel:result is None 2024-04-16T05:29:27.061 INFO:teuthology.orchestra.run.smithi122.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.7 MB/s | 548 kB 00:00 2024-04-16T05:29:27.078 INFO:teuthology.orchestra.run.smithi122.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 323 kB/s | 51 kB 00:00 2024-04-16T05:29:27.082 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:29:27.083 INFO:teuthology.orchestra.run.smithi122.stdout:Total 7.0 MB/s | 6.8 MB 00:00 2024-04-16T05:29:27.124 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-16T05:29:27.158 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-16T05:29:27.158 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-16T05:29:27.299 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-16T05:29:27.300 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-16T05:29:27.678 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-16T05:29:27.890 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T05:29:27.953 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T05:29:28.653 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T05:29:28.761 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T05:29:28.826 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T05:29:28.850 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 6/6 2024-04-16T05:29:29.775 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 6/6 2024-04-16T05:29:29.776 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 1/6 2024-04-16T05:29:29.776 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T05:29:29.776 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T05:29:29.776 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T05:29:29.776 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T05:29:30.126 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T05:29:30.127 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:30.127 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-16T05:29:30.127 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm-2:17.2.7-847.g45317f95.el9.noarch 2024-04-16T05:29:30.127 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T05:29:30.127 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T05:29:30.127 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T05:29:30.127 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T05:29:30.127 INFO:teuthology.orchestra.run.smithi122.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T05:29:30.128 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:30.128 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:29:30.472 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-fuse 2024-04-16T05:29:31.334 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:02 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:29:31.497 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:29:31.497 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:29:31.498 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-04-16T05:29:31.498 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:29:31.498 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-16T05:29:31.498 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse x86_64 2:17.2.7-847.g45317f95.el9 ceph 813 k 2024-04-16T05:29:31.498 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-04-16T05:29:31.498 INFO:teuthology.orchestra.run.smithi122.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T05:29:31.498 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:31.498 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-16T05:29:31.498 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:29:31.498 INFO:teuthology.orchestra.run.smithi122.stdout:Install 2 Packages 2024-04-16T05:29:31.498 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:31.499 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 891 k 2024-04-16T05:29:31.499 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 2.6 M 2024-04-16T05:29:31.499 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-16T05:29:31.638 INFO:teuthology.orchestra.run.smithi122.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 567 kB/s | 79 kB 00:00 2024-04-16T05:29:32.022 INFO:teuthology.orchestra.run.smithi122.stdout:(2/2): ceph-fuse-17.2.7-847.g45317f95.el9.x86_6 1.5 MB/s | 813 kB 00:00 2024-04-16T05:29:32.026 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:29:32.027 INFO:teuthology.orchestra.run.smithi122.stdout:Total 1.6 MB/s | 891 kB 00:00 2024-04-16T05:29:32.027 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-16T05:29:32.097 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-16T05:29:32.097 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-16T05:29:32.191 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-16T05:29:32.191 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-16T05:29:32.455 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-16T05:29:32.618 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T05:29:32.691 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 2/2 2024-04-16T05:29:33.568 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 2/2 2024-04-16T05:29:33.568 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/2 2024-04-16T05:29:33.900 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T05:29:33.900 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:33.900 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-16T05:29:33.900 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse-2:17.2.7-847.g45317f95.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T05:29:33.900 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:33.901 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:29:34.232 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados-devel 2024-04-16T05:29:35.129 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:06 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:29:35.289 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:29:35.289 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:29:35.289 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-04-16T05:29:35.289 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:29:35.289 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-16T05:29:35.290 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel x86_64 2:17.2.7-847.g45317f95.el9 ceph 126 k 2024-04-16T05:29:35.290 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:35.290 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-16T05:29:35.290 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:29:35.290 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-16T05:29:35.290 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:35.290 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 126 k 2024-04-16T05:29:35.290 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 448 k 2024-04-16T05:29:35.291 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-16T05:29:35.527 INFO:teuthology.orchestra.run.smithi122.stdout:librados-devel-17.2.7-847.g45317f95.el9.x86_64. 536 kB/s | 126 kB 00:00 2024-04-16T05:29:35.529 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:29:35.530 INFO:teuthology.orchestra.run.smithi122.stdout:Total 527 kB/s | 126 kB 00:00 2024-04-16T05:29:35.530 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-16T05:29:35.549 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-16T05:29:35.549 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-16T05:29:35.663 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-16T05:29:35.663 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-16T05:29:35.864 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-16T05:29:35.969 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:36.539 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:36.898 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:36.898 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:36.898 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-16T05:29:36.898 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:29:36.898 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:36.898 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:29:37.210 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs2 2024-04-16T05:29:38.086 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:09 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:29:38.191 INFO:teuthology.orchestra.run.smithi122.stdout:Package libcephfs2-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T05:29:38.244 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:29:38.246 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-16T05:29:38.246 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:29:38.316 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs-devel 2024-04-16T05:29:39.129 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:10 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:29:39.289 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:29:39.290 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:29:39.290 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-04-16T05:29:39.290 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:29:39.290 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-16T05:29:39.290 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel x86_64 2:17.2.7-847.g45317f95.el9 ceph 25 k 2024-04-16T05:29:39.290 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:39.290 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-16T05:29:39.290 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:29:39.290 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-16T05:29:39.290 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:39.290 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 25 k 2024-04-16T05:29:39.290 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 110 k 2024-04-16T05:29:39.291 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-16T05:29:39.580 INFO:teuthology.orchestra.run.smithi122.stdout:libcephfs-devel-17.2.7-847.g45317f95.el9.x86_64 86 kB/s | 25 kB 00:00 2024-04-16T05:29:39.582 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:29:39.583 INFO:teuthology.orchestra.run.smithi122.stdout:Total 85 kB/s | 25 kB 00:00 2024-04-16T05:29:39.583 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-16T05:29:39.595 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-16T05:29:39.595 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-16T05:29:39.601 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-16T05:29:39.602 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-16T05:29:39.752 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-16T05:29:39.891 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:40.345 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:40.638 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:40.638 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:40.638 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-16T05:29:40.639 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:29:40.639 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:40.639 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:29:40.846 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados2 2024-04-16T05:29:41.711 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:12 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:29:41.817 INFO:teuthology.orchestra.run.smithi122.stdout:Package librados2-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T05:29:41.872 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:29:41.873 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-16T05:29:41.873 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:29:41.938 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librbd1 2024-04-16T05:29:42.799 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:13 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:29:42.908 INFO:teuthology.orchestra.run.smithi122.stdout:Package librbd1-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T05:29:42.963 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:29:42.964 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-16T05:29:42.964 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:29:43.028 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rados 2024-04-16T05:29:43.876 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:14 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:29:43.982 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rados-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T05:29:44.038 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:29:44.039 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-16T05:29:44.039 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:29:44.111 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rgw 2024-04-16T05:29:44.958 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:15 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:29:45.063 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rgw-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T05:29:45.118 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:29:45.119 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-16T05:29:45.119 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:29:45.191 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-cephfs 2024-04-16T05:29:46.017 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:17 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:29:46.123 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-cephfs-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T05:29:46.178 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:29:46.179 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-16T05:29:46.179 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:29:46.245 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rbd 2024-04-16T05:29:47.067 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:18 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:29:47.172 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rbd-2:17.2.7-847.g45317f95.el9.x86_64 is already installed. 2024-04-16T05:29:47.225 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:29:47.226 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-04-16T05:29:47.227 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:29:47.295 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-fuse 2024-04-16T05:29:48.141 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:19 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:29:48.302 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:29:48.303 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:29:48.303 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-04-16T05:29:48.303 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:29:48.303 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-16T05:29:48.303 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse x86_64 2:17.2.7-847.g45317f95.el9 ceph 82 k 2024-04-16T05:29:48.303 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:48.303 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-16T05:29:48.303 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:29:48.303 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-16T05:29:48.304 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:48.304 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 82 k 2024-04-16T05:29:48.304 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 222 k 2024-04-16T05:29:48.304 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-16T05:29:48.645 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-fuse-17.2.7-847.g45317f95.el9.x86_64.rpm 240 kB/s | 82 kB 00:00 2024-04-16T05:29:48.646 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:29:48.647 INFO:teuthology.orchestra.run.smithi122.stdout:Total 238 kB/s | 82 kB 00:00 2024-04-16T05:29:48.647 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-16T05:29:48.670 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-16T05:29:48.670 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-16T05:29:48.772 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-16T05:29:48.772 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-16T05:29:48.990 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-16T05:29:49.117 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:49.708 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:50.030 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:50.030 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:50.030 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-16T05:29:50.030 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:29:50.030 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:50.030 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:29:50.346 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-mirror 2024-04-16T05:29:51.229 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:22 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:29:51.389 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:29:51.390 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:29:51.390 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-04-16T05:29:51.390 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:29:51.390 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-16T05:29:51.390 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror x86_64 2:17.2.7-847.g45317f95.el9 ceph 2.9 M 2024-04-16T05:29:51.390 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:51.390 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-16T05:29:51.391 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:29:51.391 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-16T05:29:51.391 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:51.391 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 2.9 M 2024-04-16T05:29:51.391 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 12 M 2024-04-16T05:29:51.391 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-16T05:29:52.206 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-mirror-17.2.7-847.g45317f95.el9.x86_64.rpm 3.6 MB/s | 2.9 MB 00:00 2024-04-16T05:29:52.208 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:29:52.209 INFO:teuthology.orchestra.run.smithi122.stdout:Total 3.6 MB/s | 2.9 MB 00:00 2024-04-16T05:29:52.210 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-16T05:29:52.240 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-16T05:29:52.240 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-16T05:29:52.368 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-16T05:29:52.368 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-16T05:29:52.887 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-16T05:29:52.949 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:53.021 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:53.022 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T05:29:53.022 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T05:29:53.022 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T05:29:53.022 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T05:29:53.022 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:54.527 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:54.527 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:54.527 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-16T05:29:54.527 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:29:54.527 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:54.527 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:29:54.854 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-nbd 2024-04-16T05:29:55.707 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:26 ago on Tue 16 Apr 2024 05:27:29 AM UTC. 2024-04-16T05:29:55.870 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-04-16T05:29:55.871 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:29:55.871 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-04-16T05:29:55.871 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:29:55.871 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-04-16T05:29:55.871 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd x86_64 2:17.2.7-847.g45317f95.el9 ceph 165 k 2024-04-16T05:29:55.871 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:55.871 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-04-16T05:29:55.871 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-04-16T05:29:55.871 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-04-16T05:29:55.871 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:55.871 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 165 k 2024-04-16T05:29:55.872 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 477 k 2024-04-16T05:29:55.872 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-04-16T05:29:56.110 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-nbd-17.2.7-847.g45317f95.el9.x86_64.rpm 695 kB/s | 165 kB 00:00 2024-04-16T05:29:56.112 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:29:56.113 INFO:teuthology.orchestra.run.smithi122.stdout:Total 684 kB/s | 165 kB 00:00 2024-04-16T05:29:56.113 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-04-16T05:29:56.139 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-04-16T05:29:56.139 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-04-16T05:29:56.237 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-04-16T05:29:56.238 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-04-16T05:29:56.455 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-04-16T05:29:56.561 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:57.105 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:57.422 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 1/1 2024-04-16T05:29:57.422 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:57.422 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-04-16T05:29:57.422 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd-2:17.2.7-847.g45317f95.el9.x86_64 2024-04-16T05:29:57.422 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:29:57.422 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-04-16T05:29:57.729 DEBUG:teuthology.parallel:result is None 2024-04-16T05:29:57.730 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T05:29:57.730 INFO:teuthology.packaging:ref: None 2024-04-16T05:29:57.730 INFO:teuthology.packaging:tag: None 2024-04-16T05:29:57.730 INFO:teuthology.packaging:branch: wip-64502-quincy-3 2024-04-16T05:29:57.730 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T05:29:57.730 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-64502-quincy-3 2024-04-16T05:29:57.960 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T05:29:58.036 INFO:teuthology.orchestra.run.smithi027.stdout:17.2.7-847.g45317f95.el9 2024-04-16T05:29:58.037 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-847.g45317f95.el9 2024-04-16T05:29:58.037 INFO:teuthology.task.install:The correct ceph version 17.2.7-847.g45317f95 is installed. 2024-04-16T05:29:58.038 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T05:29:58.038 INFO:teuthology.packaging:ref: None 2024-04-16T05:29:58.038 INFO:teuthology.packaging:tag: None 2024-04-16T05:29:58.038 INFO:teuthology.packaging:branch: wip-64502-quincy-3 2024-04-16T05:29:58.038 INFO:teuthology.packaging:sha1: 218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T05:29:58.038 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-64502-quincy-3 2024-04-16T05:29:58.181 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T05:29:58.251 INFO:teuthology.orchestra.run.smithi122.stdout:17.2.7-847.g45317f95.el9 2024-04-16T05:29:58.251 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-847.g45317f95.el9 2024-04-16T05:29:58.251 INFO:teuthology.task.install:The correct ceph version 17.2.7-847.g45317f95 is installed. 2024-04-16T05:29:58.253 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-16T05:29:58.253 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-16T05:29:58.253 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T05:29:58.372 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-16T05:29:58.372 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T05:29:58.503 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-16T05:29:58.503 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-16T05:29:58.503 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T05:29:58.616 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T05:29:58.715 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-16T05:29:58.715 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T05:29:58.815 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T05:29:58.902 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-16T05:29:58.902 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-16T05:29:58.902 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T05:29:59.002 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T05:29:59.133 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-16T05:29:59.133 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T05:29:59.231 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T05:29:59.367 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-16T05:29:59.368 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-16T05:29:59.368 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T05:29:59.458 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T05:29:59.598 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-16T05:29:59.598 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T05:29:59.694 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T05:29:59.783 INFO:teuthology.run_tasks:Running task print... 2024-04-16T05:29:59.793 INFO:teuthology.task.print:**** done install task... 2024-04-16T05:29:59.793 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-16T05:29:59.884 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '218a691417ef57c5d22a8d5cf3c5f41f7b542838'} 2024-04-16T05:29:59.885 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 2024-04-16T05:29:59.885 INFO:tasks.cephadm:Cluster fsid is 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:29:59.886 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-16T05:29:59.886 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-16T05:29:59.886 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi027': '172.21.15.27', 'mon.smithi122': '172.21.15.122'} 2024-04-16T05:29:59.886 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-16T05:29:59.886 DEBUG:teuthology.orchestra.run.smithi027:> sudo hostname $(hostname -s) 2024-04-16T05:29:59.993 DEBUG:teuthology.orchestra.run.smithi122:> sudo hostname $(hostname -s) 2024-04-16T05:30:00.092 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-04-16T05:30:00.092 DEBUG:teuthology.orchestra.run.smithi027:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-16T05:30:00.320 INFO:teuthology.orchestra.run.smithi027.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Apr 16 05:30 /home/ubuntu/cephtest/cephadm 2024-04-16T05:30:00.320 DEBUG:teuthology.orchestra.run.smithi122:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-16T05:30:00.554 INFO:teuthology.orchestra.run.smithi122.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Apr 16 05:30 /home/ubuntu/cephtest/cephadm 2024-04-16T05:30:00.554 DEBUG:teuthology.orchestra.run.smithi027:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-16T05:30:00.619 DEBUG:teuthology.orchestra.run.smithi122:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-16T05:30:00.690 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 on all hosts... 2024-04-16T05:30:00.690 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 pull 2024-04-16T05:30:00.698 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 pull 2024-04-16T05:30:01.861 INFO:teuthology.orchestra.run.smithi027.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3... 2024-04-16T05:30:01.907 INFO:teuthology.orchestra.run.smithi122.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3... 2024-04-16T05:30:38.709 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:30:38.710 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)", 2024-04-16T05:30:38.710 INFO:teuthology.orchestra.run.smithi027.stdout: "image_id": "b333581ccdf5c328f64509c4ba16a1a468c02f5eb459fd8c7d1e1ce535883bbe", 2024-04-16T05:30:38.710 INFO:teuthology.orchestra.run.smithi027.stdout: "repo_digests": [ 2024-04-16T05:30:38.710 INFO:teuthology.orchestra.run.smithi027.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77" 2024-04-16T05:30:38.710 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-04-16T05:30:38.710 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:30:41.383 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-04-16T05:30:41.383 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)", 2024-04-16T05:30:41.384 INFO:teuthology.orchestra.run.smithi122.stdout: "image_id": "b333581ccdf5c328f64509c4ba16a1a468c02f5eb459fd8c7d1e1ce535883bbe", 2024-04-16T05:30:41.384 INFO:teuthology.orchestra.run.smithi122.stdout: "repo_digests": [ 2024-04-16T05:30:41.384 INFO:teuthology.orchestra.run.smithi122.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77" 2024-04-16T05:30:41.384 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2024-04-16T05:30:41.384 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-04-16T05:30:41.426 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph 2024-04-16T05:30:41.540 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph 2024-04-16T05:30:41.648 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /etc/ceph 2024-04-16T05:30:41.761 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /etc/ceph 2024-04-16T05:30:41.856 INFO:tasks.cephadm:Writing seed config... 2024-04-16T05:30:41.857 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-04-16T05:30:41.857 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-04-16T05:30:41.857 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-04-16T05:30:41.857 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-04-16T05:30:41.857 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-04-16T05:30:41.857 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-16T05:30:41.857 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-16T05:30:41.857 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-04-16T05:30:41.857 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-04-16T05:30:41.857 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-16T05:30:41.857 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-16T05:30:41.858 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-04-16T05:30:41.858 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-16T05:30:41.858 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-16T05:30:41.858 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-16T05:30:41.858 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-16T05:30:41.858 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-16T05:30:41.858 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-16T05:30:41.858 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-16T05:30:41.858 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-16T05:30:41.858 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-16T05:30:41.858 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-16T05:30:41.858 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-16T05:30:41.858 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-04-16T05:30:41.858 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-16T05:30:41.859 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-16T05:30:41.859 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-16T05:30:41.859 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-16T05:30:41.859 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-16T05:30:41.859 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-16T05:30:41.859 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-16T05:30:41.859 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-16T05:30:41.859 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-16T05:30:41.859 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-16T05:30:41.859 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-16T05:30:41.859 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-16T05:30:41.859 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-16T05:30:41.859 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-16T05:30:41.859 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-16T05:30:41.859 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-16T05:30:41.859 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-16T05:30:41.860 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-16T05:30:41.860 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-16T05:30:41.860 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-16T05:30:41.918 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 5e3cca06-fbb2-11ee-bc8f-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-16T05:30:41.918 DEBUG:teuthology.orchestra.run.smithi027:mon.smithi027> sudo journalctl -f -n 0 -u ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@mon.smithi027.service 2024-04-16T05:30:41.964 INFO:tasks.cephadm:Bootstrapping... 2024-04-16T05:30:41.964 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 -v bootstrap --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.27 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-16T05:30:42.340 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:30:42.340 INFO:teuthology.orchestra.run.smithi027.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3', '-v', 'bootstrap', '--fsid', '5e3cca06-fbb2-11ee-bc8f-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.27', '--skip-admin-label'] 2024-04-16T05:30:42.401 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-16T05:30:42.402 INFO:teuthology.orchestra.run.smithi027.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-16T05:30:42.402 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying podman|docker is present... 2024-04-16T05:30:42.458 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-16T05:30:42.458 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying lvm2 is present... 2024-04-16T05:30:42.458 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying time synchronization is in place... 2024-04-16T05:30:42.479 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-16T05:30:42.479 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-16T05:30:42.497 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-16T05:30:42.497 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-16T05:30:42.519 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout enabled 2024-04-16T05:30:42.539 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout active 2024-04-16T05:30:42.539 INFO:teuthology.orchestra.run.smithi027.stdout:Unit chronyd.service is enabled and running 2024-04-16T05:30:42.539 INFO:teuthology.orchestra.run.smithi027.stdout:Repeating the final host check... 2024-04-16T05:30:42.601 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-16T05:30:42.601 INFO:teuthology.orchestra.run.smithi027.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-16T05:30:42.601 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl is present 2024-04-16T05:30:42.601 INFO:teuthology.orchestra.run.smithi027.stdout:lvcreate is present 2024-04-16T05:30:42.617 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-16T05:30:42.617 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-16T05:30:42.633 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-16T05:30:42.633 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-16T05:30:42.656 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout enabled 2024-04-16T05:30:42.677 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout active 2024-04-16T05:30:42.678 INFO:teuthology.orchestra.run.smithi027.stdout:Unit chronyd.service is enabled and running 2024-04-16T05:30:42.678 INFO:teuthology.orchestra.run.smithi027.stdout:Host looks OK 2024-04-16T05:30:42.678 INFO:teuthology.orchestra.run.smithi027.stdout:Cluster fsid: 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:30:42.679 INFO:teuthology.orchestra.run.smithi027.stdout:Acquiring lock 140216327693408 on /run/cephadm/5e3cca06-fbb2-11ee-bc8f-c7b262605968.lock 2024-04-16T05:30:42.679 INFO:teuthology.orchestra.run.smithi027.stdout:Lock 140216327693408 acquired on /run/cephadm/5e3cca06-fbb2-11ee-bc8f-c7b262605968.lock 2024-04-16T05:30:42.681 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying IP 172.21.15.27 port 3300 ... 2024-04-16T05:30:42.682 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying IP 172.21.15.27 port 6789 ... 2024-04-16T05:30:42.683 INFO:teuthology.orchestra.run.smithi027.stdout:Base mon IP(s) is [172.21.15.27:3300, 172.21.15.27:6789], mon addrv is [v2:172.21.15.27:3300,v1:172.21.15.27:6789] 2024-04-16T05:30:42.699 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.27 metric 100 2024-04-16T05:30:42.699 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.27 metric 100 2024-04-16T05:30:42.711 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-16T05:30:42.711 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-16T05:30:42.711 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-16T05:30:42.723 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-16T05:30:42.723 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-16T05:30:42.723 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-16T05:30:42.723 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-16T05:30:42.723 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-04-16T05:30:42.723 INFO:teuthology.orchestra.run.smithi027.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-16T05:30:42.726 INFO:teuthology.orchestra.run.smithi027.stdout:Mon IP `172.21.15.27` is in CIDR network `172.21.0.0/20` 2024-04-16T05:30:42.726 INFO:teuthology.orchestra.run.smithi027.stdout:Mon IP `172.21.15.27` is in CIDR network `172.21.0.0/20` 2024-04-16T05:30:42.726 INFO:teuthology.orchestra.run.smithi027.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-16T05:30:42.729 INFO:teuthology.orchestra.run.smithi027.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-16T05:30:42.731 INFO:teuthology.orchestra.run.smithi027.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3... 2024-04-16T05:30:49.510 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stdout b333581ccdf5c328f64509c4ba16a1a468c02f5eb459fd8c7d1e1ce535883bbe 2024-04-16T05:30:49.511 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3... 2024-04-16T05:30:49.511 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Getting image source signatures 2024-04-16T05:30:49.511 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying blob sha256:0ca96691e690249b227cbeb34b4e7fdd96c9d73da309c97947ab03def7d1129d 2024-04-16T05:30:49.511 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying blob sha256:9793b5b5582798bde394f4383d142e9f248a25e4ebfd72ee05ea9ec6251a877d 2024-04-16T05:30:49.511 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Copying config sha256:b333581ccdf5c328f64509c4ba16a1a468c02f5eb459fd8c7d1e1ce535883bbe 2024-04-16T05:30:49.511 INFO:teuthology.orchestra.run.smithi027.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-16T05:30:51.570 INFO:teuthology.orchestra.run.smithi027.stdout:ceph: stdout ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable) 2024-04-16T05:30:51.570 INFO:teuthology.orchestra.run.smithi027.stdout:Ceph version: ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable) 2024-04-16T05:30:51.570 INFO:teuthology.orchestra.run.smithi027.stdout:Extracting ceph user uid/gid from container image... 2024-04-16T05:30:52.819 INFO:teuthology.orchestra.run.smithi027.stdout:stat: stdout 167 167 2024-04-16T05:30:52.820 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial keys... 2024-04-16T05:30:53.958 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQANDR5mbxtrERAABjWSLi4uXFdzljStwCxwyg== 2024-04-16T05:30:55.215 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQAODR5mmZa/HxAAvUCB6t9qQRayIQHJ4LW5Iw== 2024-04-16T05:30:56.351 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph-authtool: stdout AQAPDR5m2Y2aKRAAzrCCXI22FSDzUDQNzSuUvQ== 2024-04-16T05:30:56.352 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial monmap... 2024-04-16T05:30:57.616 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-16T05:30:57.616 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-04-16T05:30:57.616 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:30:57.616 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-16T05:30:57.616 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool for smithi027 [v2:172.21.15.27:3300,v1:172.21.15.27:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-16T05:30:57.616 INFO:teuthology.orchestra.run.smithi027.stdout:setting min_mon_release = octopus 2024-04-16T05:30:57.616 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: set fsid to 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:30:57.616 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-16T05:30:57.616 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:30:57.617 INFO:teuthology.orchestra.run.smithi027.stdout:Creating mon... 2024-04-16T05:30:58.790 INFO:teuthology.orchestra.run.smithi027.stdout:create mon.smithi027 on 2024-04-16T05:30:59.183 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-16T05:30:59.431 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-16T05:30:59.717 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968.target → /etc/systemd/system/ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968.target. 2024-04-16T05:30:59.717 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968.target → /etc/systemd/system/ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968.target. 2024-04-16T05:31:00.031 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@mon.smithi027 2024-04-16T05:31:00.031 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to reset failed state of unit ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@mon.smithi027.service: Unit ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@mon.smithi027.service not loaded. 2024-04-16T05:31:00.350 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968.target.wants/ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@mon.smithi027.service → /etc/systemd/system/ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@.service. 2024-04-16T05:31:01.050 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:00 smithi027 podman[28818]: 2024-04-16 05:31:00.771884611 +0000 UTC m=+0.136188814 container create 9f38e22f769622e0107ce48da774ca2279eac5d8cff7944081b56a081f28a2fe (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-64502-quincy-3-45317f9, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T05:31:01.050 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:01 smithi027 podman[28818]: 2024-04-16 05:31:01.051079259 +0000 UTC m=+0.415383496 container init 9f38e22f769622e0107ce48da774ca2279eac5d8cff7944081b56a081f28a2fe (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-04-16T05:31:01.205 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-16T05:31:01.206 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-04-16T05:31:01.226 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-16T05:31:01.226 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-16T05:31:01.226 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-04-16T05:31:01.227 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to enable service . firewalld.service is not available 2024-04-16T05:31:01.228 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mon to start... 2024-04-16T05:31:01.228 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mon... 2024-04-16T05:31:01.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:01 smithi027 podman[28818]: 2024-04-16 05:31:01.062432733 +0000 UTC m=+0.426736935 container start 9f38e22f769622e0107ce48da774ca2279eac5d8cff7944081b56a081f28a2fe (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-64502-quincy-3-45317f9, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-04-16T05:31:01.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:01 smithi027 ceph-mon[28837]: mkfs 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:31:01.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:01 smithi027 ceph-mon[28837]: mon.smithi027 is new leader, mons smithi027 in quorum (ranks 0) 2024-04-16T05:31:01.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:01 smithi027 bash[28818]: 9f38e22f769622e0107ce48da774ca2279eac5d8cff7944081b56a081f28a2fe 2024-04-16T05:31:01.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:01 smithi027 systemd[1]: Started Ceph mon.smithi027 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968. 2024-04-16T05:31:02.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:02 smithi027 ceph-mon[28837]: mon.smithi027 is new leader, mons smithi027 in quorum (ranks 0) 2024-04-16T05:31:02.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:02 smithi027 ceph-mon[28837]: monmap e1: 1 mons at {smithi027=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-16T05:31:02.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:02 smithi027 ceph-mon[28837]: fsmap 2024-04-16T05:31:02.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:02 smithi027 ceph-mon[28837]: osdmap e1: 0 total, 0 up, 0 in 2024-04-16T05:31:02.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:02 smithi027 ceph-mon[28837]: mgrmap e1: no daemons active 2024-04-16T05:31:02.263 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:02 smithi027 ceph-mon[28837]: from='client.? 172.21.15.27:0/4033453872' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-16T05:31:02.551 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout cluster: 2024-04-16T05:31:02.552 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout id: 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:31:02.552 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-16T05:31:02.552 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-16T05:31:02.552 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout services: 2024-04-16T05:31:02.552 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi027 (age 0.799906s) 2024-04-16T05:31:02.553 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-16T05:31:02.553 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-16T05:31:02.559 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-16T05:31:02.559 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout data: 2024-04-16T05:31:02.559 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-16T05:31:02.559 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-16T05:31:02.559 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-16T05:31:02.559 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout pgs: 2024-04-16T05:31:02.559 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-16T05:31:02.559 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.846+0000 7fda86a00700 1 Processor -- start 2024-04-16T05:31:02.559 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.846+0000 7fda86a00700 1 -- start start 2024-04-16T05:31:02.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.847+0000 7fda86a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda801028b0 0x7fda80102cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:02.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.847+0000 7fda86a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda80103200 con 0x7fda801028b0 2024-04-16T05:31:02.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.848+0000 7fda7fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda801028b0 0x7fda80102cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:02.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.848+0000 7fda7fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda801028b0 0x7fda80102cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48024/0 (socket says 172.21.15.27:48024) 2024-04-16T05:31:02.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.848+0000 7fda7fe00700 1 -- 172.21.15.27:0/1438207329 learned_addr learned my addr 172.21.15.27:0/1438207329 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:02.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.848+0000 7fda7fe00700 1 -- 172.21.15.27:0/1438207329 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda80103340 con 0x7fda801028b0 2024-04-16T05:31:02.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.849+0000 7fda7fe00700 1 --2- 172.21.15.27:0/1438207329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda801028b0 0x7fda80102cc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fda700098e0 tx=0x7fda70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=718631327bf8ccca server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:02.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.849+0000 7fda7ea00700 1 -- 172.21.15.27:0/1438207329 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda7000b860 con 0x7fda801028b0 2024-04-16T05:31:02.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.849+0000 7fda7ea00700 1 -- 172.21.15.27:0/1438207329 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7000b9c0 con 0x7fda801028b0 2024-04-16T05:31:02.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.849+0000 7fda7ea00700 1 -- 172.21.15.27:0/1438207329 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda7000bc90 con 0x7fda801028b0 2024-04-16T05:31:02.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.850+0000 7fda86a00700 1 -- 172.21.15.27:0/1438207329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda801028b0 msgr2=0x7fda80102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:02.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.850+0000 7fda86a00700 1 --2- 172.21.15.27:0/1438207329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda801028b0 0x7fda80102cc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fda700098e0 tx=0x7fda70009250 comp rx=0 tx=0).stop 2024-04-16T05:31:02.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.850+0000 7fda86a00700 1 -- 172.21.15.27:0/1438207329 shutdown_connections 2024-04-16T05:31:02.560 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.850+0000 7fda86a00700 1 --2- 172.21.15.27:0/1438207329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda801028b0 0x7fda80102cc0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:02.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.850+0000 7fda86a00700 1 -- 172.21.15.27:0/1438207329 >> 172.21.15.27:0/1438207329 conn(0x7fda800fe070 msgr2=0x7fda80100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:02.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.850+0000 7fda86a00700 1 -- 172.21.15.27:0/1438207329 shutdown_connections 2024-04-16T05:31:02.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.850+0000 7fda86a00700 1 -- 172.21.15.27:0/1438207329 wait complete. 2024-04-16T05:31:02.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.851+0000 7fda86a00700 1 Processor -- start 2024-04-16T05:31:02.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.851+0000 7fda86a00700 1 -- start start 2024-04-16T05:31:02.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.851+0000 7fda86a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda801028b0 0x7fda80110e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:02.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.851+0000 7fda86a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda801113b0 con 0x7fda801028b0 2024-04-16T05:31:02.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.852+0000 7fda7fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda801028b0 0x7fda80110e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:02.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.852+0000 7fda7fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda801028b0 0x7fda80110e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48026/0 (socket says 172.21.15.27:48026) 2024-04-16T05:31:02.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.852+0000 7fda7fe00700 1 -- 172.21.15.27:0/4033453872 learned_addr learned my addr 172.21.15.27:0/4033453872 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:02.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.852+0000 7fda7fe00700 1 -- 172.21.15.27:0/4033453872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda700094f0 con 0x7fda801028b0 2024-04-16T05:31:02.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.852+0000 7fda7fe00700 1 --2- 172.21.15.27:0/4033453872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda801028b0 0x7fda80110e70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fda70003080 tx=0x7fda70016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:02.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.853+0000 7fda77e00700 1 -- 172.21.15.27:0/4033453872 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda7001c5c0 con 0x7fda801028b0 2024-04-16T05:31:02.561 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.853+0000 7fda77e00700 1 -- 172.21.15.27:0/4033453872 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7001c720 con 0x7fda801028b0 2024-04-16T05:31:02.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.853+0000 7fda77e00700 1 -- 172.21.15.27:0/4033453872 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda7001ca30 con 0x7fda801028b0 2024-04-16T05:31:02.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.853+0000 7fda86a00700 1 -- 172.21.15.27:0/4033453872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda801115b0 con 0x7fda801028b0 2024-04-16T05:31:02.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.853+0000 7fda86a00700 1 -- 172.21.15.27:0/4033453872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda80111fe0 con 0x7fda801028b0 2024-04-16T05:31:02.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.853+0000 7fda77e00700 1 -- 172.21.15.27:0/4033453872 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fda7001cc60 con 0x7fda801028b0 2024-04-16T05:31:02.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.853+0000 7fda77e00700 1 -- 172.21.15.27:0/4033453872 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fda7001ae00 con 0x7fda801028b0 2024-04-16T05:31:02.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.854+0000 7fda86a00700 1 -- 172.21.15.27:0/4033453872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8005f5f0 con 0x7fda801028b0 2024-04-16T05:31:02.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.858+0000 7fda77e00700 1 -- 172.21.15.27:0/4033453872 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fda7001a450 con 0x7fda801028b0 2024-04-16T05:31:02.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.938+0000 7fda86a00700 1 -- 172.21.15.27:0/4033453872 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fda80112b50 con 0x7fda801028b0 2024-04-16T05:31:02.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.938+0000 7fda77e00700 1 -- 172.21.15.27:0/4033453872 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fda70025440 con 0x7fda801028b0 2024-04-16T05:31:02.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.940+0000 7fda86a00700 1 -- 172.21.15.27:0/4033453872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda801028b0 msgr2=0x7fda80110e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:02.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.940+0000 7fda86a00700 1 --2- 172.21.15.27:0/4033453872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda801028b0 0x7fda80110e70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fda70003080 tx=0x7fda70016000 comp rx=0 tx=0).stop 2024-04-16T05:31:02.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.940+0000 7fda86a00700 1 -- 172.21.15.27:0/4033453872 shutdown_connections 2024-04-16T05:31:02.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.940+0000 7fda86a00700 1 --2- 172.21.15.27:0/4033453872 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fda801028b0 0x7fda80110e70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:02.562 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.940+0000 7fda86a00700 1 -- 172.21.15.27:0/4033453872 >> 172.21.15.27:0/4033453872 conn(0x7fda800fe070 msgr2=0x7fda80105220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:02.563 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.940+0000 7fda86a00700 1 -- 172.21.15.27:0/4033453872 shutdown_connections 2024-04-16T05:31:02.563 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:01.940+0000 7fda86a00700 1 -- 172.21.15.27:0/4033453872 wait complete. 2024-04-16T05:31:02.563 INFO:teuthology.orchestra.run.smithi027.stdout:mon is available 2024-04-16T05:31:02.563 INFO:teuthology.orchestra.run.smithi027.stdout:Assimilating anything we can from ceph.conf... 2024-04-16T05:31:03.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:03 smithi027 ceph-mon[28837]: from='client.? 172.21.15.27:0/2113384375' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-16T05:31:03.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:03 smithi027 ceph-mon[28837]: from='client.? 172.21.15.27:0/2113384375' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-16T05:31:03.952 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-16T05:31:03.953 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [global] 2024-04-16T05:31:03.953 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout fsid = 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:31:03.953 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.27:3300,v1:172.21.15.27:6789] 2024-04-16T05:31:03.953 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-16T05:31:03.953 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-16T05:31:03.953 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-16T05:31:03.953 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-16T05:31:03.953 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-16T05:31:03.953 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-16T05:31:03.953 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-16T05:31:03.953 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-16T05:31:03.953 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [osd] 2024-04-16T05:31:03.954 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-16T05:31:03.954 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-16T05:31:03.954 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-16T05:31:03.954 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-16T05:31:03.954 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-16T05:31:03.954 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.191+0000 7faaba200700 1 Processor -- start 2024-04-16T05:31:03.954 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.192+0000 7faaba200700 1 -- start start 2024-04-16T05:31:03.954 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.192+0000 7faaba200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab4105f00 0x7faab4106310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:03.954 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.192+0000 7faaba200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab41068e0 con 0x7faab4105f00 2024-04-16T05:31:03.954 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.193+0000 7faab8e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab4105f00 0x7faab4106310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:03.954 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.193+0000 7faab8e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab4105f00 0x7faab4106310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48034/0 (socket says 172.21.15.27:48034) 2024-04-16T05:31:03.954 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.193+0000 7faab8e00700 1 -- 172.21.15.27:0/2194595219 learned_addr learned my addr 172.21.15.27:0/2194595219 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:03.954 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.193+0000 7faab8e00700 1 -- 172.21.15.27:0/2194595219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab41046b0 con 0x7faab4105f00 2024-04-16T05:31:03.955 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.194+0000 7faab8e00700 1 --2- 172.21.15.27:0/2194595219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab4105f00 0x7faab4106310 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7faaa4009e50 tx=0x7faaa4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f091e78633fda0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:03.955 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.194+0000 7faab3e00700 1 -- 172.21.15.27:0/2194595219 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faaa40092e0 con 0x7faab4105f00 2024-04-16T05:31:03.955 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.194+0000 7faab3e00700 1 -- 172.21.15.27:0/2194595219 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa400ba20 con 0x7faab4105f00 2024-04-16T05:31:03.955 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.194+0000 7faab3e00700 1 -- 172.21.15.27:0/2194595219 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faaa400bd30 con 0x7faab4105f00 2024-04-16T05:31:03.955 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.195+0000 7faaba200700 1 -- 172.21.15.27:0/2194595219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab4105f00 msgr2=0x7faab4106310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:03.955 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.195+0000 7faaba200700 1 --2- 172.21.15.27:0/2194595219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab4105f00 0x7faab4106310 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7faaa4009e50 tx=0x7faaa4009840 comp rx=0 tx=0).stop 2024-04-16T05:31:03.955 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.195+0000 7faaba200700 1 -- 172.21.15.27:0/2194595219 shutdown_connections 2024-04-16T05:31:03.955 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.195+0000 7faaba200700 1 --2- 172.21.15.27:0/2194595219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab4105f00 0x7faab4106310 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:03.955 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.195+0000 7faaba200700 1 -- 172.21.15.27:0/2194595219 >> 172.21.15.27:0/2194595219 conn(0x7faab4078160 msgr2=0x7faab4078560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:03.955 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.195+0000 7faaba200700 1 -- 172.21.15.27:0/2194595219 shutdown_connections 2024-04-16T05:31:03.955 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.195+0000 7faaba200700 1 -- 172.21.15.27:0/2194595219 wait complete. 2024-04-16T05:31:03.955 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.196+0000 7faaba200700 1 Processor -- start 2024-04-16T05:31:03.955 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.196+0000 7faaba200700 1 -- start start 2024-04-16T05:31:03.955 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.197+0000 7faaba200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab4105f00 0x7faab4112e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:03.955 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.197+0000 7faaba200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab41133c0 con 0x7faab4105f00 2024-04-16T05:31:03.956 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.197+0000 7faab8e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab4105f00 0x7faab4112e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:03.956 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.197+0000 7faab8e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab4105f00 0x7faab4112e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48040/0 (socket says 172.21.15.27:48040) 2024-04-16T05:31:03.956 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.197+0000 7faab8e00700 1 -- 172.21.15.27:0/2113384375 learned_addr learned my addr 172.21.15.27:0/2113384375 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:03.956 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.197+0000 7faab8e00700 1 -- 172.21.15.27:0/2113384375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa4009ae0 con 0x7faab4105f00 2024-04-16T05:31:03.956 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.198+0000 7faab8e00700 1 --2- 172.21.15.27:0/2113384375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab4105f00 0x7faab4112e80 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7faab4105720 tx=0x7faaa4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:03.956 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.198+0000 7faab1600700 1 -- 172.21.15.27:0/2113384375 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faaa4012b90 con 0x7faab4105f00 2024-04-16T05:31:03.956 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.198+0000 7faab1600700 1 -- 172.21.15.27:0/2113384375 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa4012d10 con 0x7faab4105f00 2024-04-16T05:31:03.956 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.198+0000 7faab1600700 1 -- 172.21.15.27:0/2113384375 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faaa401e400 con 0x7faab4105f00 2024-04-16T05:31:03.956 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.198+0000 7faaba200700 1 -- 172.21.15.27:0/2113384375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab41135c0 con 0x7faab4105f00 2024-04-16T05:31:03.956 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.198+0000 7faaba200700 1 -- 172.21.15.27:0/2113384375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab4113ff0 con 0x7faab4105f00 2024-04-16T05:31:03.956 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.198+0000 7faab1600700 1 -- 172.21.15.27:0/2113384375 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7faaa4019030 con 0x7faab4105f00 2024-04-16T05:31:03.956 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.198+0000 7faab1600700 1 -- 172.21.15.27:0/2113384375 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7faaa4026400 con 0x7faab4105f00 2024-04-16T05:31:03.956 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.199+0000 7faaba200700 1 -- 172.21.15.27:0/2113384375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab405f5f0 con 0x7faab4105f00 2024-04-16T05:31:03.957 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.202+0000 7faab1600700 1 -- 172.21.15.27:0/2113384375 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7faaa4017070 con 0x7faab4105f00 2024-04-16T05:31:03.957 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.279+0000 7faaba200700 1 -- 172.21.15.27:0/2113384375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7faab410c890 con 0x7faab4105f00 2024-04-16T05:31:03.957 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.289+0000 7faab1600700 1 -- 172.21.15.27:0/2113384375 <== mon.0 v2:172.21.15.27:3300/0 7 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7faaa4038400 con 0x7faab4105f00 2024-04-16T05:31:03.957 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.289+0000 7faab1600700 1 -- 172.21.15.27:0/2113384375 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7faaa40265c0 con 0x7faab4105f00 2024-04-16T05:31:03.957 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.291+0000 7faaba200700 1 -- 172.21.15.27:0/2113384375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab4105f00 msgr2=0x7faab4112e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:03.957 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.291+0000 7faaba200700 1 --2- 172.21.15.27:0/2113384375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab4105f00 0x7faab4112e80 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7faab4105720 tx=0x7faaa4017000 comp rx=0 tx=0).stop 2024-04-16T05:31:03.957 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.291+0000 7faaba200700 1 -- 172.21.15.27:0/2113384375 shutdown_connections 2024-04-16T05:31:03.957 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.291+0000 7faaba200700 1 --2- 172.21.15.27:0/2113384375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faab4105f00 0x7faab4112e80 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:03.957 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.291+0000 7faaba200700 1 -- 172.21.15.27:0/2113384375 >> 172.21.15.27:0/2113384375 conn(0x7faab4078160 msgr2=0x7faab4107270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:03.957 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.291+0000 7faaba200700 1 -- 172.21.15.27:0/2113384375 shutdown_connections 2024-04-16T05:31:03.957 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:03.292+0000 7faaba200700 1 -- 172.21.15.27:0/2113384375 wait complete. 2024-04-16T05:31:03.957 INFO:teuthology.orchestra.run.smithi027.stdout:Generating new minimal ceph.conf... 2024-04-16T05:31:05.055 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:04 smithi027 ceph-mon[28837]: from='client.? 172.21.15.27:0/1330750318' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:31:05.657 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.572+0000 7f9881400700 1 Processor -- start 2024-04-16T05:31:05.657 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.573+0000 7f9881400700 1 -- start start 2024-04-16T05:31:05.657 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.573+0000 7f9881400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987c076c00 0x7f987c103100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:05.657 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.573+0000 7f9881400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c103640 con 0x7f987c076c00 2024-04-16T05:31:05.657 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.574+0000 7f987be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987c076c00 0x7f987c103100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:05.657 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.574+0000 7f987be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987c076c00 0x7f987c103100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57886/0 (socket says 172.21.15.27:57886) 2024-04-16T05:31:05.657 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.574+0000 7f987be00700 1 -- 172.21.15.27:0/1445861435 learned_addr learned my addr 172.21.15.27:0/1445861435 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:05.657 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.574+0000 7f987be00700 1 -- 172.21.15.27:0/1445861435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987c103780 con 0x7f987c076c00 2024-04-16T05:31:05.657 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.575+0000 7f987be00700 1 --2- 172.21.15.27:0/1445861435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987c076c00 0x7f987c103100 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f986c0098e0 tx=0x7f986c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a04934614af978a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:05.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.575+0000 7f987aa00700 1 -- 172.21.15.27:0/1445861435 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f986c00b860 con 0x7f987c076c00 2024-04-16T05:31:05.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.575+0000 7f987aa00700 1 -- 172.21.15.27:0/1445861435 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f986c00be50 con 0x7f987c076c00 2024-04-16T05:31:05.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.575+0000 7f987aa00700 1 -- 172.21.15.27:0/1445861435 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f986c010380 con 0x7f987c076c00 2024-04-16T05:31:05.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.576+0000 7f9881400700 1 -- 172.21.15.27:0/1445861435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987c076c00 msgr2=0x7f987c103100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:05.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.576+0000 7f9881400700 1 --2- 172.21.15.27:0/1445861435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987c076c00 0x7f987c103100 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f986c0098e0 tx=0x7f986c009250 comp rx=0 tx=0).stop 2024-04-16T05:31:05.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.576+0000 7f9881400700 1 -- 172.21.15.27:0/1445861435 shutdown_connections 2024-04-16T05:31:05.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.576+0000 7f9881400700 1 --2- 172.21.15.27:0/1445861435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987c076c00 0x7f987c103100 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:05.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.576+0000 7f9881400700 1 -- 172.21.15.27:0/1445861435 >> 172.21.15.27:0/1445861435 conn(0x7f987c0fcda0 msgr2=0x7f987c0ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:05.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.576+0000 7f9881400700 1 -- 172.21.15.27:0/1445861435 shutdown_connections 2024-04-16T05:31:05.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.576+0000 7f9881400700 1 -- 172.21.15.27:0/1445861435 wait complete. 2024-04-16T05:31:05.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.577+0000 7f9881400700 1 Processor -- start 2024-04-16T05:31:05.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.577+0000 7f9881400700 1 -- start start 2024-04-16T05:31:05.659 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.577+0000 7f9881400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987c076c00 0x7f987c117a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:05.659 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.577+0000 7f9881400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f987c117f60 con 0x7f987c076c00 2024-04-16T05:31:05.659 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.578+0000 7f987be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987c076c00 0x7f987c117a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:05.659 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.578+0000 7f987be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987c076c00 0x7f987c117a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57890/0 (socket says 172.21.15.27:57890) 2024-04-16T05:31:05.659 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.578+0000 7f987be00700 1 -- 172.21.15.27:0/1330750318 learned_addr learned my addr 172.21.15.27:0/1330750318 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:05.659 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.578+0000 7f987be00700 1 -- 172.21.15.27:0/1330750318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986c0094f0 con 0x7f987c076c00 2024-04-16T05:31:05.659 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.578+0000 7f987be00700 1 --2- 172.21.15.27:0/1330750318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987c076c00 0x7f987c117a20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f986c00ba90 tx=0x7f986c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:05.659 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.579+0000 7f9873e00700 1 -- 172.21.15.27:0/1330750318 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f986c01a8a0 con 0x7f987c076c00 2024-04-16T05:31:05.659 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.579+0000 7f9873e00700 1 -- 172.21.15.27:0/1330750318 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f986c01ae90 con 0x7f987c076c00 2024-04-16T05:31:05.659 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.579+0000 7f9873e00700 1 -- 172.21.15.27:0/1330750318 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f986c019ea0 con 0x7f987c076c00 2024-04-16T05:31:05.659 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.579+0000 7f9881400700 1 -- 172.21.15.27:0/1330750318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f987c118160 con 0x7f987c076c00 2024-04-16T05:31:05.659 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.579+0000 7f9881400700 1 -- 172.21.15.27:0/1330750318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f987c118cd0 con 0x7f987c076c00 2024-04-16T05:31:05.659 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.579+0000 7f9873e00700 1 -- 172.21.15.27:0/1330750318 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f986c023450 con 0x7f987c076c00 2024-04-16T05:31:05.659 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.579+0000 7f9873e00700 1 -- 172.21.15.27:0/1330750318 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f986c021780 con 0x7f987c076c00 2024-04-16T05:31:05.660 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.580+0000 7f9881400700 1 -- 172.21.15.27:0/1330750318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f987c05f5f0 con 0x7f987c076c00 2024-04-16T05:31:05.660 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.583+0000 7f9873e00700 1 -- 172.21.15.27:0/1330750318 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f986c028060 con 0x7f987c076c00 2024-04-16T05:31:05.660 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.659+0000 7f9881400700 1 -- 172.21.15.27:0/1330750318 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f987c1045c0 con 0x7f987c076c00 2024-04-16T05:31:05.660 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.660+0000 7f9873e00700 1 -- 172.21.15.27:0/1330750318 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f986c01e070 con 0x7f987c076c00 2024-04-16T05:31:05.660 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.661+0000 7f9881400700 1 -- 172.21.15.27:0/1330750318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987c076c00 msgr2=0x7f987c117a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:05.660 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.661+0000 7f9881400700 1 --2- 172.21.15.27:0/1330750318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987c076c00 0x7f987c117a20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f986c00ba90 tx=0x7f986c009250 comp rx=0 tx=0).stop 2024-04-16T05:31:05.660 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.661+0000 7f9881400700 1 -- 172.21.15.27:0/1330750318 shutdown_connections 2024-04-16T05:31:05.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.661+0000 7f9881400700 1 --2- 172.21.15.27:0/1330750318 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f987c076c00 0x7f987c117a20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:05.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.661+0000 7f9881400700 1 -- 172.21.15.27:0/1330750318 >> 172.21.15.27:0/1330750318 conn(0x7f987c0fcda0 msgr2=0x7f987c106ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:05.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.662+0000 7f9881400700 1 -- 172.21.15.27:0/1330750318 shutdown_connections 2024-04-16T05:31:05.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:04.662+0000 7f9881400700 1 -- 172.21.15.27:0/1330750318 wait complete. 2024-04-16T05:31:05.661 INFO:teuthology.orchestra.run.smithi027.stdout:Restarting the monitor... 2024-04-16T05:31:06.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:05 smithi027 systemd[1]: Stopping Ceph mon.smithi027 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T05:31:06.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:06 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027[28833]: 2024-04-16T05:31:06.176+0000 7f07e9c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi027 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T05:31:06.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:06 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027[28833]: 2024-04-16T05:31:06.176+0000 7f07e9c00700 -1 mon.smithi027@0(leader) e1 *** Got Signal Terminated *** 2024-04-16T05:31:06.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:06 smithi027 podman[29077]: 2024-04-16 05:31:06.352507842 +0000 UTC m=+0.295259720 container died 9f38e22f769622e0107ce48da774ca2279eac5d8cff7944081b56a081f28a2fe (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027, RELEASE=wip-64502-quincy-3-45317f9, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-16T05:31:07.045 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:06 smithi027 podman[29077]: 2024-04-16 05:31:06.762087857 +0000 UTC m=+0.704839760 container cleanup 9f38e22f769622e0107ce48da774ca2279eac5d8cff7944081b56a081f28a2fe (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027, org.label-schema.license=GPLv2, RELEASE=wip-64502-quincy-3-45317f9, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-16T05:31:07.045 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:06 smithi027 bash[29077]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027 2024-04-16T05:31:07.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:07 smithi027 podman[29090]: 2024-04-16 05:31:07.046050865 +0000 UTC m=+0.672325856 container remove 9f38e22f769622e0107ce48da774ca2279eac5d8cff7944081b56a081f28a2fe (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-64502-quincy-3-45317f9, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-16T05:31:07.995 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:07 smithi027 systemd[1]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@mon.smithi027.service: Deactivated successfully. 2024-04-16T05:31:07.995 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:07 smithi027 systemd[1]: Stopped Ceph mon.smithi027 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968. 2024-04-16T05:31:07.995 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:07 smithi027 systemd[1]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@mon.smithi027.service: Consumed 2.045s CPU time. 2024-04-16T05:31:07.995 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:07 smithi027 systemd[1]: Starting Ceph mon.smithi027 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T05:31:08.265 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:07 smithi027 podman[29187]: 2024-04-16 05:31:07.904124246 +0000 UTC m=+0.037339331 image pull b333581ccdf5c328f64509c4ba16a1a468c02f5eb459fd8c7d1e1ce535883bbe quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 2024-04-16T05:31:08.265 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 podman[29187]: 2024-04-16 05:31:08.013549762 +0000 UTC m=+0.146764850 container create 88575dbb8d98c7a63876cf20a632e21073d7d1bdc024885dd073ec093bc5600b (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-64502-quincy-3-45317f9, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-16T05:31:08.265 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 podman[29187]: 2024-04-16 05:31:08.267128537 +0000 UTC m=+0.400343632 container init 88575dbb8d98c7a63876cf20a632e21073d7d1bdc024885dd073ec093bc5600b (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-64502-quincy-3-45317f9, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409) 2024-04-16T05:31:08.407 INFO:teuthology.orchestra.run.smithi027.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-16T05:31:08.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 podman[29187]: 2024-04-16 05:31:08.27821695 +0000 UTC m=+0.411432011 container start 88575dbb8d98c7a63876cf20a632e21073d7d1bdc024885dd073ec093bc5600b (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-64502-quincy-3-45317f9, GIT_BRANCH=HEAD) 2024-04-16T05:31:08.559 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: set uid:gid to 167:167 (ceph:ceph) 2024-04-16T05:31:08.559 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable), process ceph-mon, pid 2 2024-04-16T05:31:08.559 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: pidfile_write: ignore empty --pid-file 2024-04-16T05:31:08.559 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: load: jerasure load: lrc 2024-04-16T05:31:08.559 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: RocksDB version: 6.15.5 2024-04-16T05:31:08.559 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-16T05:31:08.559 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Compile date Apr 12 2024 2024-04-16T05:31:08.559 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: DB SUMMARY 2024-04-16T05:31:08.559 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: DB Session ID: GTUY5JG6A5XHIHFBIWLH 2024-04-16T05:31:08.559 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: CURRENT file: CURRENT 2024-04-16T05:31:08.559 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: IDENTITY file: IDENTITY 2024-04-16T05:31:08.559 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-16T05:31:08.559 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi027/store.db dir, Total Num: 1, files: 000008.sst 2024-04-16T05:31:08.560 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi027/store.db: 000010.log size: 98290 ; 2024-04-16T05:31:08.560 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.error_if_exists: 0 2024-04-16T05:31:08.560 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.create_if_missing: 0 2024-04-16T05:31:08.560 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.paranoid_checks: 1 2024-04-16T05:31:08.560 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-16T05:31:08.560 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.env: 0x5565b645db00 2024-04-16T05:31:08.560 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.fs: Posix File System 2024-04-16T05:31:08.560 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.info_log: 0x5565e130bc00 2024-04-16T05:31:08.560 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_file_opening_threads: 16 2024-04-16T05:31:08.560 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.statistics: (nil) 2024-04-16T05:31:08.560 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.use_fsync: 0 2024-04-16T05:31:08.560 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_log_file_size: 0 2024-04-16T05:31:08.560 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-16T05:31:08.560 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-16T05:31:08.560 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.keep_log_file_num: 1000 2024-04-16T05:31:08.560 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.recycle_log_file_num: 0 2024-04-16T05:31:08.561 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.allow_fallocate: 1 2024-04-16T05:31:08.561 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.allow_mmap_reads: 0 2024-04-16T05:31:08.561 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.allow_mmap_writes: 0 2024-04-16T05:31:08.561 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.use_direct_reads: 0 2024-04-16T05:31:08.561 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-16T05:31:08.561 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.create_missing_column_families: 0 2024-04-16T05:31:08.561 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.db_log_dir: 2024-04-16T05:31:08.562 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi027/store.db 2024-04-16T05:31:08.562 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-16T05:31:08.562 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-16T05:31:08.562 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-16T05:31:08.562 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-16T05:31:08.562 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-16T05:31:08.562 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-16T05:31:08.562 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.advise_random_on_open: 1 2024-04-16T05:31:08.562 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.db_write_buffer_size: 0 2024-04-16T05:31:08.562 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.write_buffer_manager: 0x5565e2080a20 2024-04-16T05:31:08.562 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-16T05:31:08.562 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-16T05:31:08.562 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-16T05:31:08.563 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-16T05:31:08.563 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.rate_limiter: (nil) 2024-04-16T05:31:08.563 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-16T05:31:08.563 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.wal_recovery_mode: 2 2024-04-16T05:31:08.563 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.enable_thread_tracking: 0 2024-04-16T05:31:08.563 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.enable_pipelined_write: 0 2024-04-16T05:31:08.563 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.unordered_write: 0 2024-04-16T05:31:08.563 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-16T05:31:08.563 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-16T05:31:08.563 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-16T05:31:08.563 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-16T05:31:08.563 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.row_cache: None 2024-04-16T05:31:08.563 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.wal_filter: None 2024-04-16T05:31:08.564 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-16T05:31:08.564 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.allow_ingest_behind: 0 2024-04-16T05:31:08.564 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.preserve_deletes: 0 2024-04-16T05:31:08.564 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.two_write_queues: 0 2024-04-16T05:31:08.564 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.manual_wal_flush: 0 2024-04-16T05:31:08.564 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.atomic_flush: 0 2024-04-16T05:31:08.564 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-16T05:31:08.564 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-16T05:31:08.564 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-16T05:31:08.564 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.log_readahead_size: 0 2024-04-16T05:31:08.564 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-16T05:31:08.564 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.best_efforts_recovery: 0 2024-04-16T05:31:08.564 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-16T05:31:08.564 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-16T05:31:08.564 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.allow_data_in_errors: 0 2024-04-16T05:31:08.565 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.db_host_id: __hostname__ 2024-04-16T05:31:08.565 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_background_jobs: 2 2024-04-16T05:31:08.565 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_background_compactions: -1 2024-04-16T05:31:08.565 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_subcompactions: 1 2024-04-16T05:31:08.565 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-16T05:31:08.565 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-16T05:31:08.565 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-16T05:31:08.565 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_total_wal_size: 0 2024-04-16T05:31:08.565 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-16T05:31:08.565 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-16T05:31:08.565 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-16T05:31:08.565 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-16T05:31:08.565 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_open_files: -1 2024-04-16T05:31:08.565 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.bytes_per_sync: 0 2024-04-16T05:31:08.565 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-16T05:31:08.566 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-16T05:31:08.566 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.compaction_readahead_size: 0 2024-04-16T05:31:08.566 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_background_flushes: -1 2024-04-16T05:31:08.566 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Compression algorithms supported: 2024-04-16T05:31:08.566 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-16T05:31:08.566 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: kZSTD supported: 0 2024-04-16T05:31:08.566 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: kXpressCompression supported: 0 2024-04-16T05:31:08.566 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: kLZ4HCCompression supported: 1 2024-04-16T05:31:08.566 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: kZlibCompression supported: 1 2024-04-16T05:31:08.566 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: kSnappyCompression supported: 1 2024-04-16T05:31:08.566 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: kLZ4Compression supported: 1 2024-04-16T05:31:08.567 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: kBZip2Compression supported: 0 2024-04-16T05:31:08.567 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-16T05:31:08.567 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi027/store.db/MANIFEST-000009 2024-04-16T05:31:08.567 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-04-16T05:31:08.567 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-16T05:31:08.567 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.merge_operator: 2024-04-16T05:31:08.567 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.compaction_filter: None 2024-04-16T05:31:08.567 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.compaction_filter_factory: None 2024-04-16T05:31:08.567 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.sst_partitioner_factory: None 2024-04-16T05:31:08.568 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-16T05:31:08.568 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-16T05:31:08.568 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5565e1360f58) 2024-04-16T05:31:08.568 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: cache_index_and_filter_blocks: 1 2024-04-16T05:31:08.568 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-16T05:31:08.568 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-16T05:31:08.568 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: pin_top_level_index_and_filter: 1 2024-04-16T05:31:08.568 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: index_type: 0 2024-04-16T05:31:08.568 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: data_block_index_type: 0 2024-04-16T05:31:08.568 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: index_shortening: 1 2024-04-16T05:31:08.569 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-16T05:31:08.569 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: hash_index_allow_collision: 1 2024-04-16T05:31:08.569 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: checksum: 1 2024-04-16T05:31:08.569 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: no_block_cache: 0 2024-04-16T05:31:08.569 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_cache: 0x5565e1393090 2024-04-16T05:31:08.569 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_cache_name: BinnedLRUCache 2024-04-16T05:31:08.569 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_cache_options: 2024-04-16T05:31:08.569 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: capacity : 536870912 2024-04-16T05:31:08.569 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: num_shard_bits : 4 2024-04-16T05:31:08.569 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: strict_capacity_limit : 0 2024-04-16T05:31:08.569 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: high_pri_pool_ratio: 0.000 2024-04-16T05:31:08.569 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_cache_compressed: (nil) 2024-04-16T05:31:08.569 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: persistent_cache: (nil) 2024-04-16T05:31:08.569 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_size: 4096 2024-04-16T05:31:08.569 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_size_deviation: 10 2024-04-16T05:31:08.569 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_restart_interval: 16 2024-04-16T05:31:08.570 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: index_block_restart_interval: 1 2024-04-16T05:31:08.570 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: metadata_block_size: 4096 2024-04-16T05:31:08.570 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: partition_filters: 0 2024-04-16T05:31:08.570 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: use_delta_encoding: 1 2024-04-16T05:31:08.570 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-16T05:31:08.570 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: whole_key_filtering: 1 2024-04-16T05:31:08.570 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: verify_compression: 0 2024-04-16T05:31:08.570 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: read_amp_bytes_per_bit: 0 2024-04-16T05:31:08.570 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: format_version: 4 2024-04-16T05:31:08.570 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: enable_index_compression: 1 2024-04-16T05:31:08.570 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_align: 0 2024-04-16T05:31:08.570 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.write_buffer_size: 33554432 2024-04-16T05:31:08.570 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_write_buffer_number: 2 2024-04-16T05:31:08.570 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.compression: NoCompression 2024-04-16T05:31:08.570 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.bottommost_compression: Disabled 2024-04-16T05:31:08.570 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.prefix_extractor: nullptr 2024-04-16T05:31:08.571 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-16T05:31:08.571 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.num_levels: 7 2024-04-16T05:31:08.571 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-16T05:31:08.571 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-16T05:31:08.571 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-16T05:31:08.571 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-16T05:31:08.571 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-16T05:31:08.571 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-16T05:31:08.571 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-16T05:31:08.571 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-16T05:31:08.571 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-16T05:31:08.571 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-16T05:31:08.571 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-16T05:31:08.571 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.compression_opts.level: 32767 2024-04-16T05:31:08.571 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.compression_opts.strategy: 0 2024-04-16T05:31:08.572 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-16T05:31:08.572 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-16T05:31:08.572 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-16T05:31:08.572 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.compression_opts.enabled: false 2024-04-16T05:31:08.572 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-16T05:31:08.572 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-16T05:31:08.572 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-16T05:31:08.572 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.target_file_size_base: 67108864 2024-04-16T05:31:08.572 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-16T05:31:08.572 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-16T05:31:08.572 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-16T05:31:08.572 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-16T05:31:08.572 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-16T05:31:08.572 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-16T05:31:08.573 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-16T05:31:08.573 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-16T05:31:08.573 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-16T05:31:08.573 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-16T05:31:08.573 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-16T05:31:08.573 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-16T05:31:08.573 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-16T05:31:08.573 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.arena_block_size: 4194304 2024-04-16T05:31:08.573 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-16T05:31:08.573 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-16T05:31:08.573 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-16T05:31:08.573 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.disable_auto_compactions: 0 2024-04-16T05:31:08.573 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-16T05:31:08.573 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-16T05:31:08.573 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-16T05:31:08.574 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-16T05:31:08.574 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-16T05:31:08.574 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-16T05:31:08.574 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-16T05:31:08.574 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-16T05:31:08.574 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-16T05:31:08.574 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-16T05:31:08.574 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.table_properties_collectors: 2024-04-16T05:31:08.574 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.inplace_update_support: 0 2024-04-16T05:31:08.574 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-16T05:31:08.574 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-16T05:31:08.574 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-16T05:31:08.574 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-16T05:31:08.574 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.bloom_locality: 0 2024-04-16T05:31:08.575 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.max_successive_merges: 0 2024-04-16T05:31:08.575 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-16T05:31:08.575 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.paranoid_file_checks: 0 2024-04-16T05:31:08.575 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.force_consistency_checks: 1 2024-04-16T05:31:08.575 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.report_bg_io_stats: 0 2024-04-16T05:31:08.575 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.ttl: 2592000 2024-04-16T05:31:08.575 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-16T05:31:08.575 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.enable_blob_files: false 2024-04-16T05:31:08.575 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.min_blob_size: 0 2024-04-16T05:31:08.575 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.blob_file_size: 268435456 2024-04-16T05:31:08.575 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-16T05:31:08.575 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-16T05:31:08.575 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-16T05:31:08.575 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi027/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-16T05:31:08.576 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-04-16T05:31:08.576 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-04-16T05:31:08.576 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713245468330814, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-16T05:31:08.576 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-04-16T05:31:08.576 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-16T05:31:08.576 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713245468332643, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 93552, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 91636, "index_size": 276, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84172, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713245468, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "08c26cc4-d879-4639-87a0-7a5f9a4f6d9e", "db_session_id": "GTUY5JG6A5XHIHFBIWLH"}} 2024-04-16T05:31:08.576 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-04-16T05:31:08.576 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713245468333332, "job": 1, "event": "recovery_finished"} 2024-04-16T05:31:08.576 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi027/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-16T05:31:08.576 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x5565e1327180 2024-04-16T05:31:08.576 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: DB pointer 0x5565e13fc000 2024-04-16T05:31:08.576 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-04-16T05:31:08.576 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-16T05:31:08.576 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ** DB Stats ** 2024-04-16T05:31:08.577 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T05:31:08.577 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-16T05:31:08.577 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T05:31:08.577 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T05:31:08.577 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-16T05:31:08.577 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-16T05:31:08.577 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T05:31:08.577 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-16T05:31:08.577 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ** Compaction Stats [default] ** 2024-04-16T05:31:08.577 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-16T05:31:08.577 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T05:31:08.577 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: L0 2/0 93.16 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.0 0.00 0.00 1 0.001 0 0 2024-04-16T05:31:08.577 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Sum 2/0 93.16 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.0 0.00 0.00 1 0.001 0 0 2024-04-16T05:31:08.578 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.0 0.00 0.00 1 0.001 0 0 2024-04-16T05:31:08.578 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-16T05:31:08.578 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ** Compaction Stats [default] ** 2024-04-16T05:31:08.578 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-16T05:31:08.578 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T05:31:08.578 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 77.0 0.00 0.00 1 0.001 0 0 2024-04-16T05:31:08.578 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T05:31:08.578 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-16T05:31:08.578 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-16T05:31:08.578 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-16T05:31:08.578 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-16T05:31:08.578 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-16T05:31:08.578 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Cumulative compaction: 0.00 GB write, 10.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T05:31:08.578 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Interval compaction: 0.00 GB write, 10.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T05:31:08.579 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-16T05:31:08.579 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-16T05:31:08.579 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-16T05:31:08.579 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-16T05:31:08.579 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ** Compaction Stats [default] ** 2024-04-16T05:31:08.579 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-16T05:31:08.579 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T05:31:08.579 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: L0 2/0 93.16 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.0 0.00 0.00 1 0.001 0 0 2024-04-16T05:31:08.579 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Sum 2/0 93.16 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.0 0.00 0.00 1 0.001 0 0 2024-04-16T05:31:08.579 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-16T05:31:08.579 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-16T05:31:08.579 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ** Compaction Stats [default] ** 2024-04-16T05:31:08.579 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-16T05:31:08.579 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T05:31:08.579 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 77.0 0.00 0.00 1 0.001 0 0 2024-04-16T05:31:08.579 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T05:31:08.580 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-16T05:31:08.580 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-16T05:31:08.580 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-16T05:31:08.580 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-16T05:31:08.580 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-16T05:31:08.580 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Cumulative compaction: 0.00 GB write, 10.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T05:31:08.580 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T05:31:08.580 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-16T05:31:08.580 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-16T05:31:08.580 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-16T05:31:08.580 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: starting mon.smithi027 rank 0 at public addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] at bind addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi027 fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:31:08.580 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: mon.smithi027@-1(???) e1 preinit fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:31:08.580 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: mon.smithi027@-1(???).mds e1 new map 2024-04-16T05:31:08.580 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: mon.smithi027@-1(???).mds e1 print_map 2024-04-16T05:31:08.580 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: e1 2024-04-16T05:31:08.580 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:31:08.581 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:31:08.581 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: legacy client fscid: -1 2024-04-16T05:31:08.581 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-16T05:31:08.581 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: No filesystems configured 2024-04-16T05:31:08.581 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: mon.smithi027@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-16T05:31:08.581 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: mon.smithi027@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-16T05:31:08.581 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: mon.smithi027@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-16T05:31:08.581 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: mon.smithi027@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-16T05:31:08.581 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: mon.smithi027@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-16T05:31:08.581 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 bash[29187]: 88575dbb8d98c7a63876cf20a632e21073d7d1bdc024885dd073ec093bc5600b 2024-04-16T05:31:08.581 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: mon.smithi027 is new leader, mons smithi027 in quorum (ranks 0) 2024-04-16T05:31:08.581 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: monmap e1: 1 mons at {smithi027=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-16T05:31:08.581 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: fsmap 2024-04-16T05:31:08.581 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: osdmap e1: 0 total, 0 up, 0 in 2024-04-16T05:31:08.581 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 ceph-mon[29206]: mgrmap e1: no daemons active 2024-04-16T05:31:08.581 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:08 smithi027 systemd[1]: Started Ceph mon.smithi027 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968. 2024-04-16T05:31:09.828 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.031+0000 7fd55ce00700 1 Processor -- start 2024-04-16T05:31:09.834 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.031+0000 7fd55ce00700 1 -- start start 2024-04-16T05:31:09.834 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.032+0000 7fd55ce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd558077b90 0x7fd558076190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:09.834 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.032+0000 7fd55ce00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd558077fa0 con 0x7fd558077b90 2024-04-16T05:31:09.834 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.032+0000 7fd555e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd558077b90 0x7fd558076190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:09.834 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.033+0000 7fd555e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd558077b90 0x7fd558076190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57898/0 (socket says 172.21.15.27:57898) 2024-04-16T05:31:09.834 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.033+0000 7fd555e00700 1 -- 172.21.15.27:0/3426138062 learned_addr learned my addr 172.21.15.27:0/3426138062 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:09.834 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.033+0000 7fd555e00700 1 -- 172.21.15.27:0/3426138062 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5580766d0 con 0x7fd558077b90 2024-04-16T05:31:09.834 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.034+0000 7fd555e00700 1 --2- 172.21.15.27:0/3426138062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd558077b90 0x7fd558076190 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd5480098e0 tx=0x7fd548009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34db130fe2156c24 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:09.835 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.034+0000 7fd554a00700 1 -- 172.21.15.27:0/3426138062 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd54800b860 con 0x7fd558077b90 2024-04-16T05:31:09.835 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.034+0000 7fd554a00700 1 -- 172.21.15.27:0/3426138062 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fd54800be50 con 0x7fd558077b90 2024-04-16T05:31:09.835 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.034+0000 7fd554a00700 1 -- 172.21.15.27:0/3426138062 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd548010380 con 0x7fd558077b90 2024-04-16T05:31:09.835 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.035+0000 7fd55ce00700 1 -- 172.21.15.27:0/3426138062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd558077b90 msgr2=0x7fd558076190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:09.835 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.035+0000 7fd55ce00700 1 --2- 172.21.15.27:0/3426138062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd558077b90 0x7fd558076190 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd5480098e0 tx=0x7fd548009250 comp rx=0 tx=0).stop 2024-04-16T05:31:09.835 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.035+0000 7fd55ce00700 1 -- 172.21.15.27:0/3426138062 shutdown_connections 2024-04-16T05:31:09.835 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.035+0000 7fd55ce00700 1 --2- 172.21.15.27:0/3426138062 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd558077b90 0x7fd558076190 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:09.835 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.035+0000 7fd55ce00700 1 -- 172.21.15.27:0/3426138062 >> 172.21.15.27:0/3426138062 conn(0x7fd5580fe110 msgr2=0x7fd558100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:09.835 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.035+0000 7fd55ce00700 1 -- 172.21.15.27:0/3426138062 shutdown_connections 2024-04-16T05:31:09.835 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.035+0000 7fd55ce00700 1 -- 172.21.15.27:0/3426138062 wait complete. 2024-04-16T05:31:09.835 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.036+0000 7fd55ce00700 1 Processor -- start 2024-04-16T05:31:09.835 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.037+0000 7fd55ce00700 1 -- start start 2024-04-16T05:31:09.835 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.037+0000 7fd55ce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd558077b90 0x7fd558113850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:09.835 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.037+0000 7fd55ce00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd558113d90 con 0x7fd558077b90 2024-04-16T05:31:09.836 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.037+0000 7fd555e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd558077b90 0x7fd558113850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:09.836 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.037+0000 7fd555e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd558077b90 0x7fd558113850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57902/0 (socket says 172.21.15.27:57902) 2024-04-16T05:31:09.836 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.037+0000 7fd555e00700 1 -- 172.21.15.27:0/4054657220 learned_addr learned my addr 172.21.15.27:0/4054657220 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:09.836 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.038+0000 7fd555e00700 1 -- 172.21.15.27:0/4054657220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5480094f0 con 0x7fd558077b90 2024-04-16T05:31:09.836 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.038+0000 7fd555e00700 1 --2- 172.21.15.27:0/4054657220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd558077b90 0x7fd558113850 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd548006fd0 tx=0x7fd548009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:09.836 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.038+0000 7fd54e000700 1 -- 172.21.15.27:0/4054657220 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd54801a7d0 con 0x7fd558077b90 2024-04-16T05:31:09.836 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.038+0000 7fd54e000700 1 -- 172.21.15.27:0/4054657220 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fd54801adc0 con 0x7fd558077b90 2024-04-16T05:31:09.836 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.038+0000 7fd55ce00700 1 -- 172.21.15.27:0/4054657220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd558113f90 con 0x7fd558077b90 2024-04-16T05:31:09.836 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.038+0000 7fd54e000700 1 -- 172.21.15.27:0/4054657220 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd548019ea0 con 0x7fd558077b90 2024-04-16T05:31:09.836 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.039+0000 7fd55ce00700 1 -- 172.21.15.27:0/4054657220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd558114b00 con 0x7fd558077b90 2024-04-16T05:31:09.836 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.039+0000 7fd54e000700 1 -- 172.21.15.27:0/4054657220 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fd548023450 con 0x7fd558077b90 2024-04-16T05:31:09.836 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.039+0000 7fd54e000700 1 -- 172.21.15.27:0/4054657220 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd548021790 con 0x7fd558077b90 2024-04-16T05:31:09.836 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.040+0000 7fd55ce00700 1 -- 172.21.15.27:0/4054657220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd55805f5f0 con 0x7fd558077b90 2024-04-16T05:31:09.836 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.043+0000 7fd54e000700 1 -- 172.21.15.27:0/4054657220 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fd548028060 con 0x7fd558077b90 2024-04-16T05:31:09.836 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.119+0000 7fd55ce00700 1 -- 172.21.15.27:0/4054657220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fd558108b30 con 0x7fd558077b90 2024-04-16T05:31:09.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.125+0000 7fd54e000700 1 -- 172.21.15.27:0/4054657220 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd54801e070 con 0x7fd558077b90 2024-04-16T05:31:09.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.127+0000 7fd55ce00700 1 -- 172.21.15.27:0/4054657220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd558077b90 msgr2=0x7fd558113850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:09.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.127+0000 7fd55ce00700 1 --2- 172.21.15.27:0/4054657220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd558077b90 0x7fd558113850 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd548006fd0 tx=0x7fd548009250 comp rx=0 tx=0).stop 2024-04-16T05:31:09.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.127+0000 7fd55ce00700 1 -- 172.21.15.27:0/4054657220 shutdown_connections 2024-04-16T05:31:09.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.127+0000 7fd55ce00700 1 --2- 172.21.15.27:0/4054657220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd558077b90 0x7fd558113850 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:09.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.127+0000 7fd55ce00700 1 -- 172.21.15.27:0/4054657220 >> 172.21.15.27:0/4054657220 conn(0x7fd5580fe110 msgr2=0x7fd5580feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:09.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.128+0000 7fd55ce00700 1 -- 172.21.15.27:0/4054657220 shutdown_connections 2024-04-16T05:31:09.837 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:09.128+0000 7fd55ce00700 1 -- 172.21.15.27:0/4054657220 wait complete. 2024-04-16T05:31:09.837 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-16T05:31:09.837 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-16T05:31:09.837 INFO:teuthology.orchestra.run.smithi027.stdout:Creating mgr... 2024-04-16T05:31:09.837 INFO:teuthology.orchestra.run.smithi027.stdout:Verifying port 9283 ... 2024-04-16T05:31:10.158 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@mgr.smithi027.qgzypi 2024-04-16T05:31:10.159 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Failed to reset failed state of unit ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@mgr.smithi027.qgzypi.service: Unit ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@mgr.smithi027.qgzypi.service not loaded. 2024-04-16T05:31:10.441 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968.target.wants/ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@mgr.smithi027.qgzypi.service → /etc/systemd/system/ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@.service. 2024-04-16T05:31:10.473 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:10 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/4054657220' entity='client.admin' 2024-04-16T05:31:11.278 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-16T05:31:11.279 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-04-16T05:31:11.297 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-16T05:31:11.297 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-16T05:31:11.297 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-04-16T05:31:11.298 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to enable service . firewalld.service is not available 2024-04-16T05:31:11.321 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-16T05:31:11.321 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-04-16T05:31:11.341 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-16T05:31:11.341 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-16T05:31:11.342 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-04-16T05:31:11.342 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-16T05:31:11.342 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr to start... 2024-04-16T05:31:11.342 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr... 2024-04-16T05:31:12.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:12 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/965451977' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-16T05:31:12.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-16T05:31:12.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-16T05:31:12.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "5e3cca06-fbb2-11ee-bc8f-c7b262605968", 2024-04-16T05:31:12.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-04-16T05:31:12.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-16T05:31:12.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-16T05:31:12.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-16T05:31:12.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:12.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-16T05:31:12.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-16T05:31:12.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-04-16T05:31:12.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-16T05:31:12.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-16T05:31:12.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "smithi027" 2024-04-16T05:31:12.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-16T05:31:12.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-16T05:31:12.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-16T05:31:12.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T05:31:12.805 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-16T05:31:12.805 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-16T05:31:12.805 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:12.805 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-16T05:31:12.805 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T05:31:12.805 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-16T05:31:12.805 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-16T05:31:12.805 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-16T05:31:12.814 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-16T05:31:12.814 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-16T05:31:12.814 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-16T05:31:12.814 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:12.814 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-16T05:31:12.814 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-16T05:31:12.814 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-16T05:31:12.814 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-16T05:31:12.814 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-16T05:31:12.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-16T05:31:12.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-16T05:31:12.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-16T05:31:12.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-16T05:31:12.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:12.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-16T05:31:12.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T05:31:12.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-16T05:31:12.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-16T05:31:12.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:12.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-16T05:31:12.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-16T05:31:12.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-16T05:31:12.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-16T05:31:12.815 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-16T05:31:12.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-16T05:31:12.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-04-16T05:31:12.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-16T05:31:12.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T05:31:12.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:12.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-16T05:31:12.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T05:31:12.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-04-16T05:31:01.142824+0000", 2024-04-16T05:31:12.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T05:31:12.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:12.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-16T05:31:12.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-16T05:31:12.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.028+0000 7f5c9e600700 1 Processor -- start 2024-04-16T05:31:12.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.028+0000 7f5c9e600700 1 -- start start 2024-04-16T05:31:12.816 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.029+0000 7f5c9e600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c980712b0 0x7f5c98107920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:12.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.029+0000 7f5c9e600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c98103310 con 0x7f5c980712b0 2024-04-16T05:31:12.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.029+0000 7f5c97400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c980712b0 0x7f5c98107920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:12.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.030+0000 7f5c97400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c980712b0 0x7f5c98107920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57910/0 (socket says 172.21.15.27:57910) 2024-04-16T05:31:12.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.030+0000 7f5c97400700 1 -- 172.21.15.27:0/2557955707 learned_addr learned my addr 172.21.15.27:0/2557955707 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:12.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.030+0000 7f5c97400700 1 -- 172.21.15.27:0/2557955707 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c98107e60 con 0x7f5c980712b0 2024-04-16T05:31:12.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.030+0000 7f5c97400700 1 --2- 172.21.15.27:0/2557955707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c980712b0 0x7f5c98107920 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5c880098e0 tx=0x7f5c88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ec7f2fe9725d079 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:12.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.031+0000 7f5c96000700 1 -- 172.21.15.27:0/2557955707 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c8800b860 con 0x7f5c980712b0 2024-04-16T05:31:12.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.031+0000 7f5c96000700 1 -- 172.21.15.27:0/2557955707 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f5c8800be50 con 0x7f5c980712b0 2024-04-16T05:31:12.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.031+0000 7f5c96000700 1 -- 172.21.15.27:0/2557955707 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c88010380 con 0x7f5c980712b0 2024-04-16T05:31:12.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.031+0000 7f5c9e600700 1 -- 172.21.15.27:0/2557955707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c980712b0 msgr2=0x7f5c98107920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:12.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.031+0000 7f5c9e600700 1 --2- 172.21.15.27:0/2557955707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c980712b0 0x7f5c98107920 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f5c880098e0 tx=0x7f5c88009250 comp rx=0 tx=0).stop 2024-04-16T05:31:12.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.032+0000 7f5c9e600700 1 -- 172.21.15.27:0/2557955707 shutdown_connections 2024-04-16T05:31:12.817 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.032+0000 7f5c9e600700 1 --2- 172.21.15.27:0/2557955707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c980712b0 0x7f5c98107920 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:12.818 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.032+0000 7f5c9e600700 1 -- 172.21.15.27:0/2557955707 >> 172.21.15.27:0/2557955707 conn(0x7f5c980fcfd0 msgr2=0x7f5c980ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:12.818 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.032+0000 7f5c9e600700 1 -- 172.21.15.27:0/2557955707 shutdown_connections 2024-04-16T05:31:12.818 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.032+0000 7f5c9e600700 1 -- 172.21.15.27:0/2557955707 wait complete. 2024-04-16T05:31:12.818 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.033+0000 7f5c9e600700 1 Processor -- start 2024-04-16T05:31:12.818 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.033+0000 7f5c9e600700 1 -- start start 2024-04-16T05:31:12.818 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.033+0000 7f5c9e600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c980712b0 0x7f5c98113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:12.818 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.033+0000 7f5c9e600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c98113d60 con 0x7f5c980712b0 2024-04-16T05:31:12.818 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.034+0000 7f5c97400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c980712b0 0x7f5c98113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:12.818 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.034+0000 7f5c97400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c980712b0 0x7f5c98113820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57914/0 (socket says 172.21.15.27:57914) 2024-04-16T05:31:12.818 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.034+0000 7f5c97400700 1 -- 172.21.15.27:0/965451977 learned_addr learned my addr 172.21.15.27:0/965451977 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:12.818 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.034+0000 7f5c97400700 1 -- 172.21.15.27:0/965451977 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c880094f0 con 0x7f5c980712b0 2024-04-16T05:31:12.818 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.034+0000 7f5c97400700 1 --2- 172.21.15.27:0/965451977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c980712b0 0x7f5c98113820 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5c88006fd0 tx=0x7f5c88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:12.818 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.034+0000 7f5c8f400700 1 -- 172.21.15.27:0/965451977 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c8801a7d0 con 0x7f5c980712b0 2024-04-16T05:31:12.818 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.035+0000 7f5c8f400700 1 -- 172.21.15.27:0/965451977 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f5c8801adc0 con 0x7f5c980712b0 2024-04-16T05:31:12.818 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.035+0000 7f5c9e600700 1 -- 172.21.15.27:0/965451977 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c98113f60 con 0x7f5c980712b0 2024-04-16T05:31:12.819 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.035+0000 7f5c8f400700 1 -- 172.21.15.27:0/965451977 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c88019ea0 con 0x7f5c980712b0 2024-04-16T05:31:12.819 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.035+0000 7f5c9e600700 1 -- 172.21.15.27:0/965451977 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c98114ad0 con 0x7f5c980712b0 2024-04-16T05:31:12.819 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.035+0000 7f5c8f400700 1 -- 172.21.15.27:0/965451977 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f5c88023450 con 0x7f5c980712b0 2024-04-16T05:31:12.819 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.035+0000 7f5c8f400700 1 -- 172.21.15.27:0/965451977 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5c8802b6a0 con 0x7f5c980712b0 2024-04-16T05:31:12.819 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.036+0000 7f5c9e600700 1 -- 172.21.15.27:0/965451977 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c9805f5f0 con 0x7f5c980712b0 2024-04-16T05:31:12.819 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.039+0000 7f5c8f400700 1 -- 172.21.15.27:0/965451977 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f5c88028060 con 0x7f5c980712b0 2024-04-16T05:31:12.819 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.117+0000 7f5c9e600700 1 -- 172.21.15.27:0/965451977 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5c98114d80 con 0x7f5c980712b0 2024-04-16T05:31:12.819 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.118+0000 7f5c8f400700 1 -- 172.21.15.27:0/965451977 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f5c8801e070 con 0x7f5c980712b0 2024-04-16T05:31:12.819 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.119+0000 7f5c9e600700 1 -- 172.21.15.27:0/965451977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c980712b0 msgr2=0x7f5c98113820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:12.819 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.119+0000 7f5c9e600700 1 --2- 172.21.15.27:0/965451977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c980712b0 0x7f5c98113820 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5c88006fd0 tx=0x7f5c88009250 comp rx=0 tx=0).stop 2024-04-16T05:31:12.819 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.119+0000 7f5c9e600700 1 -- 172.21.15.27:0/965451977 shutdown_connections 2024-04-16T05:31:12.819 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.119+0000 7f5c9e600700 1 --2- 172.21.15.27:0/965451977 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c980712b0 0x7f5c98113820 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:12.819 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.119+0000 7f5c9e600700 1 -- 172.21.15.27:0/965451977 >> 172.21.15.27:0/965451977 conn(0x7f5c980fcfd0 msgr2=0x7f5c980fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:12.819 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.120+0000 7f5c9e600700 1 -- 172.21.15.27:0/965451977 shutdown_connections 2024-04-16T05:31:12.819 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:12.120+0000 7f5c9e600700 1 -- 172.21.15.27:0/965451977 wait complete. 2024-04-16T05:31:12.820 INFO:teuthology.orchestra.run.smithi027.stdout:mgr not available, waiting (1/15)... 2024-04-16T05:31:15.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:15 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/2181642110' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-16T05:31:16.406 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-16T05:31:16.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-16T05:31:16.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "5e3cca06-fbb2-11ee-bc8f-c7b262605968", 2024-04-16T05:31:16.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-04-16T05:31:16.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-16T05:31:16.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-16T05:31:16.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-16T05:31:16.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:16.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-16T05:31:16.407 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-16T05:31:16.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-04-16T05:31:16.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-16T05:31:16.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-16T05:31:16.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "smithi027" 2024-04-16T05:31:16.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-16T05:31:16.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-16T05:31:16.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-16T05:31:16.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T05:31:16.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-16T05:31:16.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-16T05:31:16.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:16.408 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-16T05:31:16.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T05:31:16.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-16T05:31:16.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-16T05:31:16.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-16T05:31:16.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-16T05:31:16.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-16T05:31:16.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-16T05:31:16.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:16.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-16T05:31:16.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-16T05:31:16.409 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-16T05:31:16.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-16T05:31:16.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-16T05:31:16.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-16T05:31:16.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-16T05:31:16.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-16T05:31:16.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-16T05:31:16.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:16.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-16T05:31:16.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T05:31:16.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-16T05:31:16.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-16T05:31:16.410 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:16.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-16T05:31:16.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-16T05:31:16.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-16T05:31:16.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-16T05:31:16.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-16T05:31:16.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-16T05:31:16.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-04-16T05:31:16.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-16T05:31:16.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T05:31:16.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:16.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-16T05:31:16.411 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T05:31:16.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-04-16T05:31:01.142824+0000", 2024-04-16T05:31:16.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T05:31:16.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:16.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-16T05:31:16.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-16T05:31:16.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.458+0000 7ffb55c00700 1 Processor -- start 2024-04-16T05:31:16.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.458+0000 7ffb55c00700 1 -- start start 2024-04-16T05:31:16.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.459+0000 7ffb55c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb50102900 0x7ffb50102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:16.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.459+0000 7ffb55c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb50103250 con 0x7ffb50102900 2024-04-16T05:31:16.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.459+0000 7ffb4ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb50102900 0x7ffb50102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:16.412 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.459+0000 7ffb4ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb50102900 0x7ffb50102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49140/0 (socket says 172.21.15.27:49140) 2024-04-16T05:31:16.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.459+0000 7ffb4ea00700 1 -- 172.21.15.27:0/791752934 learned_addr learned my addr 172.21.15.27:0/791752934 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:16.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.460+0000 7ffb4ea00700 1 -- 172.21.15.27:0/791752934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb50103390 con 0x7ffb50102900 2024-04-16T05:31:16.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.460+0000 7ffb4ea00700 1 --2- 172.21.15.27:0/791752934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb50102900 0x7ffb50102d10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ffb400098e0 tx=0x7ffb40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae49e72832e3816 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:16.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.460+0000 7ffb4d600700 1 -- 172.21.15.27:0/791752934 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb4000b860 con 0x7ffb50102900 2024-04-16T05:31:16.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.461+0000 7ffb4d600700 1 -- 172.21.15.27:0/791752934 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7ffb4000be50 con 0x7ffb50102900 2024-04-16T05:31:16.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.461+0000 7ffb4d600700 1 -- 172.21.15.27:0/791752934 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb40010380 con 0x7ffb50102900 2024-04-16T05:31:16.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.461+0000 7ffb55c00700 1 -- 172.21.15.27:0/791752934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb50102900 msgr2=0x7ffb50102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:16.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.461+0000 7ffb55c00700 1 --2- 172.21.15.27:0/791752934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb50102900 0x7ffb50102d10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ffb400098e0 tx=0x7ffb40009250 comp rx=0 tx=0).stop 2024-04-16T05:31:16.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.461+0000 7ffb55c00700 1 -- 172.21.15.27:0/791752934 shutdown_connections 2024-04-16T05:31:16.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.461+0000 7ffb55c00700 1 --2- 172.21.15.27:0/791752934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb50102900 0x7ffb50102d10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:16.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.461+0000 7ffb55c00700 1 -- 172.21.15.27:0/791752934 >> 172.21.15.27:0/791752934 conn(0x7ffb500fe0a0 msgr2=0x7ffb501004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:16.413 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.462+0000 7ffb55c00700 1 -- 172.21.15.27:0/791752934 shutdown_connections 2024-04-16T05:31:16.414 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.462+0000 7ffb55c00700 1 -- 172.21.15.27:0/791752934 wait complete. 2024-04-16T05:31:16.414 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.462+0000 7ffb55c00700 1 Processor -- start 2024-04-16T05:31:16.414 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.463+0000 7ffb55c00700 1 -- start start 2024-04-16T05:31:16.414 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.463+0000 7ffb55c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb50102900 0x7ffb50113890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:16.414 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.463+0000 7ffb55c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb50113dd0 con 0x7ffb50102900 2024-04-16T05:31:16.414 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.463+0000 7ffb4ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb50102900 0x7ffb50113890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:16.414 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.463+0000 7ffb4ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb50102900 0x7ffb50113890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49146/0 (socket says 172.21.15.27:49146) 2024-04-16T05:31:16.414 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.463+0000 7ffb4ea00700 1 -- 172.21.15.27:0/2181642110 learned_addr learned my addr 172.21.15.27:0/2181642110 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:16.414 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.464+0000 7ffb4ea00700 1 -- 172.21.15.27:0/2181642110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb400094f0 con 0x7ffb50102900 2024-04-16T05:31:16.414 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.464+0000 7ffb4ea00700 1 --2- 172.21.15.27:0/2181642110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb50102900 0x7ffb50113890 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ffb40000c00 tx=0x7ffb40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:16.414 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.464+0000 7ffb46a00700 1 -- 172.21.15.27:0/2181642110 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb4001a8a0 con 0x7ffb50102900 2024-04-16T05:31:16.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.464+0000 7ffb46a00700 1 -- 172.21.15.27:0/2181642110 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7ffb4001ae90 con 0x7ffb50102900 2024-04-16T05:31:16.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.464+0000 7ffb46a00700 1 -- 172.21.15.27:0/2181642110 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb40019e70 con 0x7ffb50102900 2024-04-16T05:31:16.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.464+0000 7ffb55c00700 1 -- 172.21.15.27:0/2181642110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb50113fd0 con 0x7ffb50102900 2024-04-16T05:31:16.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.465+0000 7ffb55c00700 1 -- 172.21.15.27:0/2181642110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb50114af0 con 0x7ffb50102900 2024-04-16T05:31:16.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.466+0000 7ffb46a00700 1 -- 172.21.15.27:0/2181642110 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7ffb40019730 con 0x7ffb50102900 2024-04-16T05:31:16.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.466+0000 7ffb55c00700 1 -- 172.21.15.27:0/2181642110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb5005f5f0 con 0x7ffb50102900 2024-04-16T05:31:16.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.466+0000 7ffb46a00700 1 -- 172.21.15.27:0/2181642110 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ffb400237a0 con 0x7ffb50102900 2024-04-16T05:31:16.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.468+0000 7ffb46a00700 1 -- 172.21.15.27:0/2181642110 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7ffb40028060 con 0x7ffb50102900 2024-04-16T05:31:16.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.547+0000 7ffb55c00700 1 -- 172.21.15.27:0/2181642110 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ffb501046e0 con 0x7ffb50102900 2024-04-16T05:31:16.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.548+0000 7ffb46a00700 1 -- 172.21.15.27:0/2181642110 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7ffb4001e070 con 0x7ffb50102900 2024-04-16T05:31:16.415 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.549+0000 7ffb55c00700 1 -- 172.21.15.27:0/2181642110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb50102900 msgr2=0x7ffb50113890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:16.416 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.549+0000 7ffb55c00700 1 --2- 172.21.15.27:0/2181642110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb50102900 0x7ffb50113890 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ffb40000c00 tx=0x7ffb40009250 comp rx=0 tx=0).stop 2024-04-16T05:31:16.416 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.549+0000 7ffb55c00700 1 -- 172.21.15.27:0/2181642110 shutdown_connections 2024-04-16T05:31:16.416 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.549+0000 7ffb55c00700 1 --2- 172.21.15.27:0/2181642110 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb50102900 0x7ffb50113890 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:16.416 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.549+0000 7ffb55c00700 1 -- 172.21.15.27:0/2181642110 >> 172.21.15.27:0/2181642110 conn(0x7ffb500fe0a0 msgr2=0x7ffb500fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:16.416 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.550+0000 7ffb55c00700 1 -- 172.21.15.27:0/2181642110 shutdown_connections 2024-04-16T05:31:16.416 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:15.550+0000 7ffb55c00700 1 -- 172.21.15.27:0/2181642110 wait complete. 2024-04-16T05:31:16.416 INFO:teuthology.orchestra.run.smithi027.stdout:mgr not available, waiting (2/15)... 2024-04-16T05:31:19.441 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:19 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/2514553506' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-16T05:31:19.788 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-16T05:31:19.788 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-16T05:31:19.788 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "5e3cca06-fbb2-11ee-bc8f-c7b262605968", 2024-04-16T05:31:19.788 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-04-16T05:31:19.788 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-16T05:31:19.788 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-16T05:31:19.788 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-16T05:31:19.789 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:19.789 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-16T05:31:19.789 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-16T05:31:19.789 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-04-16T05:31:19.789 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-16T05:31:19.789 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-16T05:31:19.789 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "smithi027" 2024-04-16T05:31:19.793 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-16T05:31:19.793 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-16T05:31:19.793 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-16T05:31:19.793 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T05:31:19.793 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-16T05:31:19.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-16T05:31:19.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:19.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-16T05:31:19.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T05:31:19.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-16T05:31:19.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-16T05:31:19.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-16T05:31:19.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-16T05:31:19.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-16T05:31:19.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-16T05:31:19.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:19.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-16T05:31:19.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-16T05:31:19.794 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-16T05:31:19.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-16T05:31:19.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-16T05:31:19.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-16T05:31:19.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-16T05:31:19.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-16T05:31:19.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-16T05:31:19.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:19.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-16T05:31:19.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T05:31:19.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-16T05:31:19.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-16T05:31:19.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:19.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-16T05:31:19.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-16T05:31:19.795 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-16T05:31:19.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-16T05:31:19.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-16T05:31:19.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-16T05:31:19.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-04-16T05:31:19.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-16T05:31:19.796 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T05:31:19.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:19.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-16T05:31:19.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T05:31:19.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-04-16T05:31:01.142824+0000", 2024-04-16T05:31:19.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T05:31:19.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:19.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-16T05:31:19.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-16T05:31:19.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.011+0000 7fa320600700 1 Processor -- start 2024-04-16T05:31:19.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.012+0000 7fa320600700 1 -- start start 2024-04-16T05:31:19.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.012+0000 7fa320600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa318102900 0x7fa318102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:19.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.012+0000 7fa320600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa318103250 con 0x7fa318102900 2024-04-16T05:31:19.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.013+0000 7fa31dc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa318102900 0x7fa318102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:19.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.013+0000 7fa31dc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa318102900 0x7fa318102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49152/0 (socket says 172.21.15.27:49152) 2024-04-16T05:31:19.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.013+0000 7fa31dc00700 1 -- 172.21.15.27:0/2998772665 learned_addr learned my addr 172.21.15.27:0/2998772665 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:19.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.013+0000 7fa31dc00700 1 -- 172.21.15.27:0/2998772665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa318103390 con 0x7fa318102900 2024-04-16T05:31:19.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.013+0000 7fa31dc00700 1 --2- 172.21.15.27:0/2998772665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa318102900 0x7fa318102d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa30c0098e0 tx=0x7fa30c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d710c75e40e269e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:19.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.014+0000 7fa317e00700 1 -- 172.21.15.27:0/2998772665 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa30c00b860 con 0x7fa318102900 2024-04-16T05:31:19.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.014+0000 7fa317e00700 1 -- 172.21.15.27:0/2998772665 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fa30c00be50 con 0x7fa318102900 2024-04-16T05:31:19.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.014+0000 7fa317e00700 1 -- 172.21.15.27:0/2998772665 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa30c010380 con 0x7fa318102900 2024-04-16T05:31:19.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.015+0000 7fa320600700 1 -- 172.21.15.27:0/2998772665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa318102900 msgr2=0x7fa318102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:19.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.015+0000 7fa320600700 1 --2- 172.21.15.27:0/2998772665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa318102900 0x7fa318102d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa30c0098e0 tx=0x7fa30c009250 comp rx=0 tx=0).stop 2024-04-16T05:31:19.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.015+0000 7fa320600700 1 -- 172.21.15.27:0/2998772665 shutdown_connections 2024-04-16T05:31:19.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.015+0000 7fa320600700 1 --2- 172.21.15.27:0/2998772665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa318102900 0x7fa318102d10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:19.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.015+0000 7fa320600700 1 -- 172.21.15.27:0/2998772665 >> 172.21.15.27:0/2998772665 conn(0x7fa3180fe0a0 msgr2=0x7fa3181004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:19.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.015+0000 7fa320600700 1 -- 172.21.15.27:0/2998772665 shutdown_connections 2024-04-16T05:31:19.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.015+0000 7fa320600700 1 -- 172.21.15.27:0/2998772665 wait complete. 2024-04-16T05:31:19.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.016+0000 7fa320600700 1 Processor -- start 2024-04-16T05:31:19.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.016+0000 7fa320600700 1 -- start start 2024-04-16T05:31:19.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.016+0000 7fa320600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa318102900 0x7fa318113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:19.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.016+0000 7fa320600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa318113d60 con 0x7fa318102900 2024-04-16T05:31:19.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.017+0000 7fa31dc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa318102900 0x7fa318113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:19.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.017+0000 7fa31dc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa318102900 0x7fa318113820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49160/0 (socket says 172.21.15.27:49160) 2024-04-16T05:31:19.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.017+0000 7fa31dc00700 1 -- 172.21.15.27:0/2514553506 learned_addr learned my addr 172.21.15.27:0/2514553506 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:19.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.017+0000 7fa31dc00700 1 -- 172.21.15.27:0/2514553506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa30c0094f0 con 0x7fa318102900 2024-04-16T05:31:19.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.017+0000 7fa31dc00700 1 --2- 172.21.15.27:0/2514553506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa318102900 0x7fa318113820 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa30c000c00 tx=0x7fa30c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:19.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.018+0000 7fa315600700 1 -- 172.21.15.27:0/2514553506 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa30c01a8a0 con 0x7fa318102900 2024-04-16T05:31:19.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.018+0000 7fa315600700 1 -- 172.21.15.27:0/2514553506 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fa30c01ae90 con 0x7fa318102900 2024-04-16T05:31:19.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.018+0000 7fa320600700 1 -- 172.21.15.27:0/2514553506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa318113f60 con 0x7fa318102900 2024-04-16T05:31:19.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.018+0000 7fa315600700 1 -- 172.21.15.27:0/2514553506 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa30c019e70 con 0x7fa318102900 2024-04-16T05:31:19.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.019+0000 7fa320600700 1 -- 172.21.15.27:0/2514553506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa318114ad0 con 0x7fa318102900 2024-04-16T05:31:19.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.019+0000 7fa315600700 1 -- 172.21.15.27:0/2514553506 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fa30c019730 con 0x7fa318102900 2024-04-16T05:31:19.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.019+0000 7fa315600700 1 -- 172.21.15.27:0/2514553506 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa30c0237a0 con 0x7fa318102900 2024-04-16T05:31:19.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.020+0000 7fa320600700 1 -- 172.21.15.27:0/2514553506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa31805f5f0 con 0x7fa318102900 2024-04-16T05:31:19.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.022+0000 7fa315600700 1 -- 172.21.15.27:0/2514553506 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fa30c01e070 con 0x7fa318102900 2024-04-16T05:31:19.805 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.101+0000 7fa320600700 1 -- 172.21.15.27:0/2514553506 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa3181046e0 con 0x7fa318102900 2024-04-16T05:31:19.805 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.102+0000 7fa315600700 1 -- 172.21.15.27:0/2514553506 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7fa30c01acd0 con 0x7fa318102900 2024-04-16T05:31:19.805 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.104+0000 7fa320600700 1 -- 172.21.15.27:0/2514553506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa318102900 msgr2=0x7fa318113820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:19.805 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.104+0000 7fa320600700 1 --2- 172.21.15.27:0/2514553506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa318102900 0x7fa318113820 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa30c000c00 tx=0x7fa30c009250 comp rx=0 tx=0).stop 2024-04-16T05:31:19.805 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.104+0000 7fa320600700 1 -- 172.21.15.27:0/2514553506 shutdown_connections 2024-04-16T05:31:19.805 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.104+0000 7fa320600700 1 --2- 172.21.15.27:0/2514553506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa318102900 0x7fa318113820 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:19.805 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.104+0000 7fa320600700 1 -- 172.21.15.27:0/2514553506 >> 172.21.15.27:0/2514553506 conn(0x7fa3180fe0a0 msgr2=0x7fa3180feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:19.805 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.104+0000 7fa320600700 1 -- 172.21.15.27:0/2514553506 shutdown_connections 2024-04-16T05:31:19.805 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:19.104+0000 7fa320600700 1 -- 172.21.15.27:0/2514553506 wait complete. 2024-04-16T05:31:19.805 INFO:teuthology.orchestra.run.smithi027.stdout:mgr not available, waiting (3/15)... 2024-04-16T05:31:20.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:20 smithi027 ceph-mon[29206]: Activating manager daemon smithi027.qgzypi 2024-04-16T05:31:20.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:20 smithi027 ceph-mon[29206]: mgrmap e2: smithi027.qgzypi(active, starting, since 0.00997827s) 2024-04-16T05:31:20.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:20 smithi027 ceph-mon[29206]: from='mgr.14100 172.21.15.27:0/1035482550' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T05:31:20.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:20 smithi027 ceph-mon[29206]: from='mgr.14100 172.21.15.27:0/1035482550' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T05:31:20.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:20 smithi027 ceph-mon[29206]: from='mgr.14100 172.21.15.27:0/1035482550' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T05:31:20.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:20 smithi027 ceph-mon[29206]: from='mgr.14100 172.21.15.27:0/1035482550' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-16T05:31:20.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:20 smithi027 ceph-mon[29206]: from='mgr.14100 172.21.15.27:0/1035482550' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr metadata", "who": "smithi027.qgzypi", "id": "smithi027.qgzypi"}]: dispatch 2024-04-16T05:31:20.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:20 smithi027 ceph-mon[29206]: Manager daemon smithi027.qgzypi is now available 2024-04-16T05:31:21.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:21 smithi027 ceph-mon[29206]: from='mgr.14100 172.21.15.27:0/1035482550' entity='mgr.smithi027.qgzypi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.qgzypi/mirror_snapshot_schedule"}]: dispatch 2024-04-16T05:31:21.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:21 smithi027 ceph-mon[29206]: from='mgr.14100 172.21.15.27:0/1035482550' entity='mgr.smithi027.qgzypi' 2024-04-16T05:31:21.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:21 smithi027 ceph-mon[29206]: from='mgr.14100 172.21.15.27:0/1035482550' entity='mgr.smithi027.qgzypi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.qgzypi/trash_purge_schedule"}]: dispatch 2024-04-16T05:31:21.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:21 smithi027 ceph-mon[29206]: from='mgr.14100 172.21.15.27:0/1035482550' entity='mgr.smithi027.qgzypi' 2024-04-16T05:31:21.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:21 smithi027 ceph-mon[29206]: from='mgr.14100 172.21.15.27:0/1035482550' entity='mgr.smithi027.qgzypi' 2024-04-16T05:31:21.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:21 smithi027 ceph-mon[29206]: mgrmap e3: smithi027.qgzypi(active, since 1.0208s) 2024-04-16T05:31:23.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:23 smithi027 ceph-mon[29206]: mgrmap e4: smithi027.qgzypi(active, since 2s) 2024-04-16T05:31:23.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:23 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/4284645440' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-16T05:31:23.660 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-16T05:31:23.660 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-16T05:31:23.660 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsid": "5e3cca06-fbb2-11ee-bc8f-c7b262605968", 2024-04-16T05:31:23.660 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "health": { 2024-04-16T05:31:23.660 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-16T05:31:23.660 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-16T05:31:23.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-16T05:31:23.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:23.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-16T05:31:23.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-16T05:31:23.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 0 2024-04-16T05:31:23.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-16T05:31:23.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-16T05:31:23.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "smithi027" 2024-04-16T05:31:23.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-16T05:31:23.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-16T05:31:23.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-16T05:31:23.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T05:31:23.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-04-16T05:31:23.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-16T05:31:23.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:23.662 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-16T05:31:23.662 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T05:31:23.662 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-16T05:31:23.662 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-16T05:31:23.662 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-16T05:31:23.671 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-16T05:31:23.671 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-16T05:31:23.671 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-16T05:31:23.671 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:23.671 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-16T05:31:23.671 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-16T05:31:23.671 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-16T05:31:23.671 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-16T05:31:23.671 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-16T05:31:23.671 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-16T05:31:23.671 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-16T05:31:23.672 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-16T05:31:23.672 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-16T05:31:23.672 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:23.672 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-16T05:31:23.672 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T05:31:23.672 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-16T05:31:23.672 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-16T05:31:23.672 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:23.672 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-16T05:31:23.672 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-16T05:31:23.672 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-16T05:31:23.672 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-16T05:31:23.672 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-16T05:31:23.672 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-16T05:31:23.672 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "restful" 2024-04-16T05:31:23.672 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ], 2024-04-16T05:31:23.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T05:31:23.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:23.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-16T05:31:23.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-16T05:31:23.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "modified": "2024-04-16T05:31:01.142824+0000", 2024-04-16T05:31:23.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-16T05:31:23.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout }, 2024-04-16T05:31:23.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-16T05:31:23.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-16T05:31:23.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.409+0000 7f16f5c00700 1 Processor -- start 2024-04-16T05:31:23.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.409+0000 7f16f5c00700 1 -- start start 2024-04-16T05:31:23.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.409+0000 7f16f5c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f007ca40 0x7f16f007ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:23.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.409+0000 7f16f5c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f007d420 con 0x7f16f007ca40 2024-04-16T05:31:23.673 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.410+0000 7f16eea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f007ca40 0x7f16f007ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:23.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.410+0000 7f16eea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f007ca40 0x7f16f007ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49218/0 (socket says 172.21.15.27:49218) 2024-04-16T05:31:23.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.410+0000 7f16eea00700 1 -- 172.21.15.27:0/895484219 learned_addr learned my addr 172.21.15.27:0/895484219 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:23.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.410+0000 7f16eea00700 1 -- 172.21.15.27:0/895484219 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f007d560 con 0x7f16f007ca40 2024-04-16T05:31:23.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.411+0000 7f16eea00700 1 --2- 172.21.15.27:0/895484219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f007ca40 0x7f16f007ce50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f16e00098e0 tx=0x7f16e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b47bd52009ddc192 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:23.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.411+0000 7f16ee000700 1 -- 172.21.15.27:0/895484219 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16e000b860 con 0x7f16f007ca40 2024-04-16T05:31:23.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.411+0000 7f16ee000700 1 -- 172.21.15.27:0/895484219 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f16e000be50 con 0x7f16f007ca40 2024-04-16T05:31:23.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.411+0000 7f16ee000700 1 -- 172.21.15.27:0/895484219 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16e0010380 con 0x7f16f007ca40 2024-04-16T05:31:23.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.412+0000 7f16f5c00700 1 -- 172.21.15.27:0/895484219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f007ca40 msgr2=0x7f16f007ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:23.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.412+0000 7f16f5c00700 1 --2- 172.21.15.27:0/895484219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f007ca40 0x7f16f007ce50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f16e00098e0 tx=0x7f16e0009250 comp rx=0 tx=0).stop 2024-04-16T05:31:23.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.412+0000 7f16f5c00700 1 -- 172.21.15.27:0/895484219 shutdown_connections 2024-04-16T05:31:23.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.412+0000 7f16f5c00700 1 --2- 172.21.15.27:0/895484219 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f007ca40 0x7f16f007ce50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:23.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.412+0000 7f16f5c00700 1 -- 172.21.15.27:0/895484219 >> 172.21.15.27:0/895484219 conn(0x7f16f0078350 msgr2=0x7f16f007a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:23.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.412+0000 7f16f5c00700 1 -- 172.21.15.27:0/895484219 shutdown_connections 2024-04-16T05:31:23.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.412+0000 7f16f5c00700 1 -- 172.21.15.27:0/895484219 wait complete. 2024-04-16T05:31:23.674 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.413+0000 7f16f5c00700 1 Processor -- start 2024-04-16T05:31:23.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.413+0000 7f16f5c00700 1 -- start start 2024-04-16T05:31:23.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.413+0000 7f16f5c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f007ca40 0x7f16f0117be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:23.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.413+0000 7f16f5c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f0118120 con 0x7f16f007ca40 2024-04-16T05:31:23.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.414+0000 7f16eea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f007ca40 0x7f16f0117be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:23.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.414+0000 7f16eea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f007ca40 0x7f16f0117be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49228/0 (socket says 172.21.15.27:49228) 2024-04-16T05:31:23.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.414+0000 7f16eea00700 1 -- 172.21.15.27:0/4284645440 learned_addr learned my addr 172.21.15.27:0/4284645440 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:23.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.414+0000 7f16eea00700 1 -- 172.21.15.27:0/4284645440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e00094f0 con 0x7f16f007ca40 2024-04-16T05:31:23.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.414+0000 7f16eea00700 1 --2- 172.21.15.27:0/4284645440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f007ca40 0x7f16f0117be0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f16e0000c00 tx=0x7f16e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:23.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.415+0000 7f16e6a00700 1 -- 172.21.15.27:0/4284645440 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16e001a8a0 con 0x7f16f007ca40 2024-04-16T05:31:23.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.415+0000 7f16e6a00700 1 -- 172.21.15.27:0/4284645440 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f16e001ae90 con 0x7f16f007ca40 2024-04-16T05:31:23.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.415+0000 7f16e6a00700 1 -- 172.21.15.27:0/4284645440 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16e0019e70 con 0x7f16f007ca40 2024-04-16T05:31:23.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.415+0000 7f16f5c00700 1 -- 172.21.15.27:0/4284645440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f0118320 con 0x7f16f007ca40 2024-04-16T05:31:23.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.415+0000 7f16f5c00700 1 -- 172.21.15.27:0/4284645440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f0118e90 con 0x7f16f007ca40 2024-04-16T05:31:23.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.416+0000 7f16e6a00700 1 -- 172.21.15.27:0/4284645440 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 43365+0+0 (secure 0 0 0) 0x7f16e0019730 con 0x7f16f007ca40 2024-04-16T05:31:23.675 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.416+0000 7f16f5c00700 1 -- 172.21.15.27:0/4284645440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f005f5f0 con 0x7f16f007ca40 2024-04-16T05:31:23.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.416+0000 7f16e6a00700 1 --2- 172.21.15.27:0/4284645440 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f16d4036850 0x7f16d4038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:23.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.416+0000 7f16e6a00700 1 -- 172.21.15.27:0/4284645440 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f16e004a020 con 0x7f16f007ca40 2024-04-16T05:31:23.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.417+0000 7f16e7e00700 1 --2- 172.21.15.27:0/4284645440 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f16d4036850 0x7f16d4038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:23.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.417+0000 7f16e7e00700 1 --2- 172.21.15.27:0/4284645440 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f16d4036850 0x7f16d4038d00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f16d8009990 tx=0x7f16d8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:23.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.423+0000 7f16e6a00700 1 -- 172.21.15.27:0/4284645440 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f16e0021d30 con 0x7f16f007ca40 2024-04-16T05:31:23.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.727+0000 7f16f5c00700 1 -- 172.21.15.27:0/4284645440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f16f004bed0 con 0x7f16f007ca40 2024-04-16T05:31:23.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.728+0000 7f16e6a00700 1 -- 172.21.15.27:0/4284645440 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f16e001e070 con 0x7f16f007ca40 2024-04-16T05:31:23.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.730+0000 7f16f5c00700 1 -- 172.21.15.27:0/4284645440 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f16d4036850 msgr2=0x7f16d4038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:23.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.730+0000 7f16f5c00700 1 --2- 172.21.15.27:0/4284645440 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f16d4036850 0x7f16d4038d00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f16d8009990 tx=0x7f16d8006e40 comp rx=0 tx=0).stop 2024-04-16T05:31:23.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.731+0000 7f16f5c00700 1 -- 172.21.15.27:0/4284645440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f007ca40 msgr2=0x7f16f0117be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:23.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.731+0000 7f16f5c00700 1 --2- 172.21.15.27:0/4284645440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f007ca40 0x7f16f0117be0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f16e0000c00 tx=0x7f16e0009250 comp rx=0 tx=0).stop 2024-04-16T05:31:23.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.731+0000 7f16f5c00700 1 -- 172.21.15.27:0/4284645440 shutdown_connections 2024-04-16T05:31:23.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.731+0000 7f16f5c00700 1 --2- 172.21.15.27:0/4284645440 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f16d4036850 0x7f16d4038d00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:23.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.731+0000 7f16f5c00700 1 --2- 172.21.15.27:0/4284645440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16f007ca40 0x7f16f0117be0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:23.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.731+0000 7f16f5c00700 1 -- 172.21.15.27:0/4284645440 >> 172.21.15.27:0/4284645440 conn(0x7f16f0078350 msgr2=0x7f16f0079dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:23.676 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.731+0000 7f16f5c00700 1 -- 172.21.15.27:0/4284645440 shutdown_connections 2024-04-16T05:31:23.677 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:22.731+0000 7f16f5c00700 1 -- 172.21.15.27:0/4284645440 wait complete. 2024-04-16T05:31:23.677 INFO:teuthology.orchestra.run.smithi027.stdout:mgr is available 2024-04-16T05:31:24.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:24 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1600801715' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-16T05:31:25.179 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-16T05:31:25.179 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [global] 2024-04-16T05:31:25.179 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout fsid = 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:31:25.179 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-16T05:31:25.179 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-16T05:31:25.179 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-16T05:31:25.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-16T05:31:25.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-16T05:31:25.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-16T05:31:25.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-16T05:31:25.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 2024-04-16T05:31:25.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout [osd] 2024-04-16T05:31:25.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-16T05:31:25.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-16T05:31:25.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-16T05:31:25.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-16T05:31:25.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-16T05:31:25.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.267+0000 7f2694a00700 1 Processor -- start 2024-04-16T05:31:25.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.267+0000 7f2694a00700 1 -- start start 2024-04-16T05:31:25.180 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.267+0000 7f2694a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f269010a300 0x7f269010a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:25.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.267+0000 7f2694a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269010ace0 con 0x7f269010a300 2024-04-16T05:31:25.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.268+0000 7f268f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f269010a300 0x7f269010a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:25.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.268+0000 7f268f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f269010a300 0x7f269010a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58996/0 (socket says 172.21.15.27:58996) 2024-04-16T05:31:25.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.268+0000 7f268f400700 1 -- 172.21.15.27:0/2823485601 learned_addr learned my addr 172.21.15.27:0/2823485601 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:25.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.268+0000 7f268f400700 1 -- 172.21.15.27:0/2823485601 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f269007c4e0 con 0x7f269010a300 2024-04-16T05:31:25.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.269+0000 7f268f400700 1 --2- 172.21.15.27:0/2823485601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f269010a300 0x7f269010a710 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2680009e50 tx=0x7f2680009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ccf1130c9ab018 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:25.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.269+0000 7f268ea00700 1 -- 172.21.15.27:0/2823485601 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26800092e0 con 0x7f269010a300 2024-04-16T05:31:25.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.269+0000 7f268ea00700 1 -- 172.21.15.27:0/2823485601 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f2680012430 con 0x7f269010a300 2024-04-16T05:31:25.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.269+0000 7f268ea00700 1 -- 172.21.15.27:0/2823485601 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26800075c0 con 0x7f269010a300 2024-04-16T05:31:25.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.270+0000 7f2694a00700 1 -- 172.21.15.27:0/2823485601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f269010a300 msgr2=0x7f269010a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:25.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.270+0000 7f2694a00700 1 --2- 172.21.15.27:0/2823485601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f269010a300 0x7f269010a710 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2680009e50 tx=0x7f2680009840 comp rx=0 tx=0).stop 2024-04-16T05:31:25.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.270+0000 7f2694a00700 1 -- 172.21.15.27:0/2823485601 shutdown_connections 2024-04-16T05:31:25.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.270+0000 7f2694a00700 1 --2- 172.21.15.27:0/2823485601 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f269010a300 0x7f269010a710 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:25.181 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.270+0000 7f2694a00700 1 -- 172.21.15.27:0/2823485601 >> 172.21.15.27:0/2823485601 conn(0x7f2690078010 msgr2=0x7f269007a420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:25.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.270+0000 7f2694a00700 1 -- 172.21.15.27:0/2823485601 shutdown_connections 2024-04-16T05:31:25.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.270+0000 7f2694a00700 1 -- 172.21.15.27:0/2823485601 wait complete. 2024-04-16T05:31:25.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.271+0000 7f2694a00700 1 Processor -- start 2024-04-16T05:31:25.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.271+0000 7f2694a00700 1 -- start start 2024-04-16T05:31:25.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.272+0000 7f2694a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f269010a300 0x7f2690117a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:25.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.272+0000 7f2694a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2690117f60 con 0x7f269010a300 2024-04-16T05:31:25.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.272+0000 7f268f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f269010a300 0x7f2690117a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:25.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.272+0000 7f268f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f269010a300 0x7f2690117a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59012/0 (socket says 172.21.15.27:59012) 2024-04-16T05:31:25.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.272+0000 7f268f400700 1 -- 172.21.15.27:0/1600801715 learned_addr learned my addr 172.21.15.27:0/1600801715 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:25.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.272+0000 7f268f400700 1 -- 172.21.15.27:0/1600801715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2680009ae0 con 0x7f269010a300 2024-04-16T05:31:25.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.273+0000 7f268f400700 1 --2- 172.21.15.27:0/1600801715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f269010a300 0x7f2690117a20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2680012ed0 tx=0x7f2680009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:25.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.273+0000 7f2687e00700 1 -- 172.21.15.27:0/1600801715 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f268000bc30 con 0x7f269010a300 2024-04-16T05:31:25.182 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.273+0000 7f2687e00700 1 -- 172.21.15.27:0/1600801715 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f268000bd90 con 0x7f269010a300 2024-04-16T05:31:25.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.273+0000 7f2694a00700 1 -- 172.21.15.27:0/1600801715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2690118160 con 0x7f269010a300 2024-04-16T05:31:25.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.273+0000 7f2687e00700 1 -- 172.21.15.27:0/1600801715 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f268001e9f0 con 0x7f269010a300 2024-04-16T05:31:25.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.273+0000 7f2694a00700 1 -- 172.21.15.27:0/1600801715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2690118cd0 con 0x7f269010a300 2024-04-16T05:31:25.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.274+0000 7f2687e00700 1 -- 172.21.15.27:0/1600801715 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 43365+0+0 (secure 0 0 0) 0x7f268001ec60 con 0x7f269010a300 2024-04-16T05:31:25.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.274+0000 7f2694a00700 1 -- 172.21.15.27:0/1600801715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f269005f5f0 con 0x7f269010a300 2024-04-16T05:31:25.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.275+0000 7f2687e00700 1 --2- 172.21.15.27:0/1600801715 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f2674036850 0x7f2674038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:25.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.275+0000 7f2687e00700 1 -- 172.21.15.27:0/1600801715 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f268004a770 con 0x7f269010a300 2024-04-16T05:31:25.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.275+0000 7f2686a00700 1 --2- 172.21.15.27:0/1600801715 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f2674036850 0x7f2674038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:25.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.276+0000 7f2686a00700 1 --2- 172.21.15.27:0/1600801715 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f2674036850 0x7f2674038d00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2678009990 tx=0x7f2678006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:25.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.280+0000 7f2687e00700 1 -- 172.21.15.27:0/1600801715 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f2680007d50 con 0x7f269010a300 2024-04-16T05:31:25.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.498+0000 7f2694a00700 1 -- 172.21.15.27:0/1600801715 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f269010d0d0 con 0x7f269010a300 2024-04-16T05:31:25.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.501+0000 7f2687e00700 1 -- 172.21.15.27:0/1600801715 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f2680017070 con 0x7f269010a300 2024-04-16T05:31:25.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.504+0000 7f2694a00700 1 -- 172.21.15.27:0/1600801715 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f2674036850 msgr2=0x7f2674038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:25.183 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.504+0000 7f2694a00700 1 --2- 172.21.15.27:0/1600801715 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f2674036850 0x7f2674038d00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2678009990 tx=0x7f2678006e40 comp rx=0 tx=0).stop 2024-04-16T05:31:25.184 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.504+0000 7f2694a00700 1 -- 172.21.15.27:0/1600801715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f269010a300 msgr2=0x7f2690117a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:25.184 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.504+0000 7f2694a00700 1 --2- 172.21.15.27:0/1600801715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f269010a300 0x7f2690117a20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2680012ed0 tx=0x7f2680009840 comp rx=0 tx=0).stop 2024-04-16T05:31:25.184 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.504+0000 7f2694a00700 1 -- 172.21.15.27:0/1600801715 shutdown_connections 2024-04-16T05:31:25.184 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.504+0000 7f2694a00700 1 --2- 172.21.15.27:0/1600801715 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f2674036850 0x7f2674038d00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:25.184 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.504+0000 7f2694a00700 1 --2- 172.21.15.27:0/1600801715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f269010a300 0x7f2690117a20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:25.184 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.504+0000 7f2694a00700 1 -- 172.21.15.27:0/1600801715 >> 172.21.15.27:0/1600801715 conn(0x7f2690078010 msgr2=0x7f2690078a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:25.184 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.504+0000 7f2694a00700 1 -- 172.21.15.27:0/1600801715 shutdown_connections 2024-04-16T05:31:25.184 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:24.505+0000 7f2694a00700 1 -- 172.21.15.27:0/1600801715 wait complete. 2024-04-16T05:31:25.184 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling cephadm module... 2024-04-16T05:31:26.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:26 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1152300288' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-16T05:31:26.932 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.819+0000 7f24c5000700 1 Processor -- start 2024-04-16T05:31:26.933 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.819+0000 7f24c5000700 1 -- start start 2024-04-16T05:31:26.933 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.820+0000 7f24c5000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24c0071240 0x7f24c01079a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:26.933 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.820+0000 7f24c5000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c0071650 con 0x7f24c0071240 2024-04-16T05:31:26.933 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.820+0000 7f24bde00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24c0071240 0x7f24c01079a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:26.933 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.820+0000 7f24bde00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24c0071240 0x7f24c01079a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59026/0 (socket says 172.21.15.27:59026) 2024-04-16T05:31:26.934 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.820+0000 7f24bde00700 1 -- 172.21.15.27:0/221842576 learned_addr learned my addr 172.21.15.27:0/221842576 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:26.935 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.821+0000 7f24bde00700 1 -- 172.21.15.27:0/221842576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c0103390 con 0x7f24c0071240 2024-04-16T05:31:26.935 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.821+0000 7f24bde00700 1 --2- 172.21.15.27:0/221842576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24c0071240 0x7f24c01079a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f24b00098e0 tx=0x7f24b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48bb24e449faa71b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:26.935 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.821+0000 7f24bca00700 1 -- 172.21.15.27:0/221842576 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24b000b860 con 0x7f24c0071240 2024-04-16T05:31:26.935 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.821+0000 7f24bca00700 1 -- 172.21.15.27:0/221842576 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f24b000be50 con 0x7f24c0071240 2024-04-16T05:31:26.935 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.821+0000 7f24bca00700 1 -- 172.21.15.27:0/221842576 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24b0010380 con 0x7f24c0071240 2024-04-16T05:31:26.935 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.822+0000 7f24c5000700 1 -- 172.21.15.27:0/221842576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24c0071240 msgr2=0x7f24c01079a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:26.935 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.822+0000 7f24c5000700 1 --2- 172.21.15.27:0/221842576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24c0071240 0x7f24c01079a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f24b00098e0 tx=0x7f24b0009250 comp rx=0 tx=0).stop 2024-04-16T05:31:26.937 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.822+0000 7f24c5000700 1 -- 172.21.15.27:0/221842576 shutdown_connections 2024-04-16T05:31:26.937 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.822+0000 7f24c5000700 1 --2- 172.21.15.27:0/221842576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24c0071240 0x7f24c01079a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:26.937 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.822+0000 7f24c5000700 1 -- 172.21.15.27:0/221842576 >> 172.21.15.27:0/221842576 conn(0x7f24c00fcfd0 msgr2=0x7f24c00ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:26.937 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.822+0000 7f24c5000700 1 -- 172.21.15.27:0/221842576 shutdown_connections 2024-04-16T05:31:26.937 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.822+0000 7f24c5000700 1 -- 172.21.15.27:0/221842576 wait complete. 2024-04-16T05:31:26.937 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.823+0000 7f24c5000700 1 Processor -- start 2024-04-16T05:31:26.937 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.823+0000 7f24c5000700 1 -- start start 2024-04-16T05:31:26.938 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.824+0000 7f24c5000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24c0071240 0x7f24c0113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:26.938 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.824+0000 7f24c5000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c0113d60 con 0x7f24c0071240 2024-04-16T05:31:26.938 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.824+0000 7f24bde00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24c0071240 0x7f24c0113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:26.938 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.824+0000 7f24bde00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24c0071240 0x7f24c0113820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59040/0 (socket says 172.21.15.27:59040) 2024-04-16T05:31:26.938 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.824+0000 7f24bde00700 1 -- 172.21.15.27:0/1152300288 learned_addr learned my addr 172.21.15.27:0/1152300288 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:26.938 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.824+0000 7f24bde00700 1 -- 172.21.15.27:0/1152300288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b00094f0 con 0x7f24c0071240 2024-04-16T05:31:26.938 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.825+0000 7f24bde00700 1 --2- 172.21.15.27:0/1152300288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24c0071240 0x7f24c0113820 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f24b0006fd0 tx=0x7f24b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:26.938 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.825+0000 7f24b6000700 1 -- 172.21.15.27:0/1152300288 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24b001a7d0 con 0x7f24c0071240 2024-04-16T05:31:26.938 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.825+0000 7f24b6000700 1 -- 172.21.15.27:0/1152300288 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f24b001adc0 con 0x7f24c0071240 2024-04-16T05:31:26.938 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.825+0000 7f24c5000700 1 -- 172.21.15.27:0/1152300288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c0113fc0 con 0x7f24c0071240 2024-04-16T05:31:26.938 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.825+0000 7f24b6000700 1 -- 172.21.15.27:0/1152300288 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24b0019ea0 con 0x7f24c0071240 2024-04-16T05:31:26.940 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.825+0000 7f24c5000700 1 -- 172.21.15.27:0/1152300288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c0114ad0 con 0x7f24c0071240 2024-04-16T05:31:26.941 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.826+0000 7f24c5000700 1 -- 172.21.15.27:0/1152300288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c005f5f0 con 0x7f24c0071240 2024-04-16T05:31:26.941 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.826+0000 7f24b6000700 1 -- 172.21.15.27:0/1152300288 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 43365+0+0 (secure 0 0 0) 0x7f24b0019730 con 0x7f24c0071240 2024-04-16T05:31:26.941 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.827+0000 7f24b6000700 1 --2- 172.21.15.27:0/1152300288 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f24a4036850 0x7f24a4038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:26.941 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.827+0000 7f24b6000700 1 -- 172.21.15.27:0/1152300288 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f24b0049fb0 con 0x7f24c0071240 2024-04-16T05:31:26.941 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.827+0000 7f24bd400700 1 --2- 172.21.15.27:0/1152300288 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f24a4036850 0x7f24a4038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:26.941 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.828+0000 7f24bd400700 1 --2- 172.21.15.27:0/1152300288 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f24a4036850 0x7f24a4038d00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f24ac009990 tx=0x7f24ac006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:26.941 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:25.832+0000 7f24b6000700 1 -- 172.21.15.27:0/1152300288 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f24b00193c0 con 0x7f24c0071240 2024-04-16T05:31:26.941 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:26.104+0000 7f24c5000700 1 -- 172.21.15.27:0/1152300288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f24c004bed0 con 0x7f24c0071240 2024-04-16T05:31:26.941 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:26.161+0000 7f24b6000700 1 -- 172.21.15.27:0/1152300288 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 5) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f24b001fd40 con 0x7f24c0071240 2024-04-16T05:31:26.941 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:26.164+0000 7f24b6000700 1 -- 172.21.15.27:0/1152300288 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f24b00206e0 con 0x7f24c0071240 2024-04-16T05:31:26.941 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:26.166+0000 7f24c5000700 1 -- 172.21.15.27:0/1152300288 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f24a4036850 msgr2=0x7f24a4038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:26.941 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:26.166+0000 7f24c5000700 1 --2- 172.21.15.27:0/1152300288 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f24a4036850 0x7f24a4038d00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f24ac009990 tx=0x7f24ac006e40 comp rx=0 tx=0).stop 2024-04-16T05:31:26.941 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:26.166+0000 7f24c5000700 1 -- 172.21.15.27:0/1152300288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24c0071240 msgr2=0x7f24c0113820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:26.942 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:26.166+0000 7f24c5000700 1 --2- 172.21.15.27:0/1152300288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24c0071240 0x7f24c0113820 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f24b0006fd0 tx=0x7f24b0009250 comp rx=0 tx=0).stop 2024-04-16T05:31:26.942 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:26.166+0000 7f24c5000700 1 -- 172.21.15.27:0/1152300288 shutdown_connections 2024-04-16T05:31:26.942 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:26.166+0000 7f24c5000700 1 --2- 172.21.15.27:0/1152300288 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f24a4036850 0x7f24a4038d00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:26.942 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:26.166+0000 7f24c5000700 1 --2- 172.21.15.27:0/1152300288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24c0071240 0x7f24c0113820 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:26.942 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:26.166+0000 7f24c5000700 1 -- 172.21.15.27:0/1152300288 >> 172.21.15.27:0/1152300288 conn(0x7f24c00fcfd0 msgr2=0x7f24c00fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:26.942 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:26.166+0000 7f24c5000700 1 -- 172.21.15.27:0/1152300288 shutdown_connections 2024-04-16T05:31:26.942 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:26.167+0000 7f24c5000700 1 -- 172.21.15.27:0/1152300288 wait complete. 2024-04-16T05:31:27.290 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:27 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1152300288' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-16T05:31:27.291 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:27 smithi027 ceph-mon[29206]: mgrmap e5: smithi027.qgzypi(active, since 6s) 2024-04-16T05:31:28.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:28 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1855628141' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-16T05:31:28.568 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-16T05:31:28.568 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-16T05:31:28.568 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-16T05:31:28.568 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "active_name": "smithi027.qgzypi", 2024-04-16T05:31:28.568 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-16T05:31:28.568 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-16T05:31:28.568 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.569+0000 7faa0ce00700 1 Processor -- start 2024-04-16T05:31:28.571 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.569+0000 7faa0ce00700 1 -- start start 2024-04-16T05:31:28.571 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.569+0000 7faa0ce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa081017d0 0x7faa08101be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:28.571 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.569+0000 7faa0ce00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa08102120 con 0x7faa081017d0 2024-04-16T05:31:28.571 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.570+0000 7faa05e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa081017d0 0x7faa08101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:28.571 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.570+0000 7faa05e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa081017d0 0x7faa08101be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59072/0 (socket says 172.21.15.27:59072) 2024-04-16T05:31:28.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.570+0000 7faa05e00700 1 -- 172.21.15.27:0/796213253 learned_addr learned my addr 172.21.15.27:0/796213253 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:28.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.570+0000 7faa05e00700 1 -- 172.21.15.27:0/796213253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa08102260 con 0x7faa081017d0 2024-04-16T05:31:28.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.571+0000 7faa05e00700 1 --2- 172.21.15.27:0/796213253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa081017d0 0x7faa08101be0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa9f80098e0 tx=0x7fa9f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bd73c12eff8261 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:28.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.571+0000 7faa04a00700 1 -- 172.21.15.27:0/796213253 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9f800b860 con 0x7faa081017d0 2024-04-16T05:31:28.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.571+0000 7faa04a00700 1 -- 172.21.15.27:0/796213253 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fa9f800be50 con 0x7faa081017d0 2024-04-16T05:31:28.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.571+0000 7faa04a00700 1 -- 172.21.15.27:0/796213253 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9f8010380 con 0x7faa081017d0 2024-04-16T05:31:28.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.572+0000 7faa0ce00700 1 -- 172.21.15.27:0/796213253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa081017d0 msgr2=0x7faa08101be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:28.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.572+0000 7faa0ce00700 1 --2- 172.21.15.27:0/796213253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa081017d0 0x7faa08101be0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa9f80098e0 tx=0x7fa9f8009250 comp rx=0 tx=0).stop 2024-04-16T05:31:28.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.572+0000 7faa0ce00700 1 -- 172.21.15.27:0/796213253 shutdown_connections 2024-04-16T05:31:28.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.572+0000 7faa0ce00700 1 --2- 172.21.15.27:0/796213253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa081017d0 0x7faa08101be0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:28.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.572+0000 7faa0ce00700 1 -- 172.21.15.27:0/796213253 >> 172.21.15.27:0/796213253 conn(0x7faa080fcfd0 msgr2=0x7faa080ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:28.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.572+0000 7faa0ce00700 1 -- 172.21.15.27:0/796213253 shutdown_connections 2024-04-16T05:31:28.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.572+0000 7faa0ce00700 1 -- 172.21.15.27:0/796213253 wait complete. 2024-04-16T05:31:28.572 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.573+0000 7faa0ce00700 1 Processor -- start 2024-04-16T05:31:28.573 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.574+0000 7faa0ce00700 1 -- start start 2024-04-16T05:31:28.573 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.574+0000 7faa0ce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa081017d0 0x7faa08107ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:28.573 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.574+0000 7faa0ce00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa08107ff0 con 0x7faa081017d0 2024-04-16T05:31:28.573 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.574+0000 7faa05e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa081017d0 0x7faa08107ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:28.573 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.574+0000 7faa05e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa081017d0 0x7faa08107ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59080/0 (socket says 172.21.15.27:59080) 2024-04-16T05:31:28.573 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.574+0000 7faa05e00700 1 -- 172.21.15.27:0/1855628141 learned_addr learned my addr 172.21.15.27:0/1855628141 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:28.573 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.575+0000 7faa05e00700 1 -- 172.21.15.27:0/1855628141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f80094f0 con 0x7faa081017d0 2024-04-16T05:31:28.573 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.575+0000 7faa05e00700 1 --2- 172.21.15.27:0/1855628141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa081017d0 0x7faa08107ab0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8000c00 tx=0x7fa9f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:28.576 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.575+0000 7fa9fe000700 1 -- 172.21.15.27:0/1855628141 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9f801a8a0 con 0x7faa081017d0 2024-04-16T05:31:28.576 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.575+0000 7fa9fe000700 1 -- 172.21.15.27:0/1855628141 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fa9f801ae90 con 0x7faa081017d0 2024-04-16T05:31:28.576 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.575+0000 7fa9fe000700 1 -- 172.21.15.27:0/1855628141 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9f8019e70 con 0x7faa081017d0 2024-04-16T05:31:28.576 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.575+0000 7faa0ce00700 1 -- 172.21.15.27:0/1855628141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa08106060 con 0x7faa081017d0 2024-04-16T05:31:28.576 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.576+0000 7faa0ce00700 1 -- 172.21.15.27:0/1855628141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa08106550 con 0x7faa081017d0 2024-04-16T05:31:28.576 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.577+0000 7fa9fe000700 1 -- 172.21.15.27:0/1855628141 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fa9f8018420 con 0x7faa081017d0 2024-04-16T05:31:28.576 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.577+0000 7faa0ce00700 1 -- 172.21.15.27:0/1855628141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa0804bed0 con 0x7faa081017d0 2024-04-16T05:31:28.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.577+0000 7fa9fe000700 1 --2- 172.21.15.27:0/1855628141 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7fa9ec0368a0 0x7fa9ec038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:28.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.577+0000 7fa9fe000700 1 -- 172.21.15.27:0/1855628141 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa9f804a030 con 0x7faa081017d0 2024-04-16T05:31:28.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.577+0000 7faa05400700 1 -- 172.21.15.27:0/1855628141 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7fa9ec0368a0 msgr2=0x7fa9ec038d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2173136120 2024-04-16T05:31:28.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.577+0000 7faa05400700 1 --2- 172.21.15.27:0/1855628141 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7fa9ec0368a0 0x7fa9ec038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T05:31:28.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.582+0000 7fa9fe000700 1 -- 172.21.15.27:0/1855628141 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa9f8010560 con 0x7faa081017d0 2024-04-16T05:31:28.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.778+0000 7faa05400700 1 -- 172.21.15.27:0/1855628141 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7fa9ec0368a0 msgr2=0x7fa9ec038d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2173136120 2024-04-16T05:31:28.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.778+0000 7faa05400700 1 --2- 172.21.15.27:0/1855628141 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7fa9ec0368a0 0x7fa9ec038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T05:31:28.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.854+0000 7faa0ce00700 1 -- 172.21.15.27:0/1855628141 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7faa0810b6c0 con 0x7faa081017d0 2024-04-16T05:31:28.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.854+0000 7fa9fe000700 1 -- 172.21.15.27:0/1855628141 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fa9f801e020 con 0x7faa081017d0 2024-04-16T05:31:28.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.857+0000 7faa0ce00700 1 -- 172.21.15.27:0/1855628141 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7fa9ec0368a0 msgr2=0x7fa9ec038d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T05:31:28.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.857+0000 7faa0ce00700 1 --2- 172.21.15.27:0/1855628141 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7fa9ec0368a0 0x7fa9ec038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:28.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.857+0000 7faa0ce00700 1 -- 172.21.15.27:0/1855628141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa081017d0 msgr2=0x7faa08107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:28.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.857+0000 7faa0ce00700 1 --2- 172.21.15.27:0/1855628141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa081017d0 0x7faa08107ab0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8000c00 tx=0x7fa9f8009250 comp rx=0 tx=0).stop 2024-04-16T05:31:28.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.857+0000 7faa0ce00700 1 -- 172.21.15.27:0/1855628141 shutdown_connections 2024-04-16T05:31:28.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.857+0000 7faa0ce00700 1 --2- 172.21.15.27:0/1855628141 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7fa9ec0368a0 0x7fa9ec038d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:28.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.857+0000 7faa0ce00700 1 --2- 172.21.15.27:0/1855628141 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faa081017d0 0x7faa08107ab0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:28.578 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.857+0000 7faa0ce00700 1 -- 172.21.15.27:0/1855628141 >> 172.21.15.27:0/1855628141 conn(0x7faa080fcfd0 msgr2=0x7faa080fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:28.578 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.858+0000 7faa0ce00700 1 -- 172.21.15.27:0/1855628141 shutdown_connections 2024-04-16T05:31:28.578 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:27.858+0000 7faa0ce00700 1 -- 172.21.15.27:0/1855628141 wait complete. 2024-04-16T05:31:28.578 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for the mgr to restart... 2024-04-16T05:31:28.578 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr epoch 5... 2024-04-16T05:31:35.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:35 smithi027 ceph-mon[29206]: Active manager daemon smithi027.qgzypi restarted 2024-04-16T05:31:35.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:35 smithi027 ceph-mon[29206]: Activating manager daemon smithi027.qgzypi 2024-04-16T05:31:35.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:35 smithi027 ceph-mon[29206]: osdmap e2: 0 total, 0 up, 0 in 2024-04-16T05:31:35.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:35 smithi027 ceph-mon[29206]: mgrmap e6: smithi027.qgzypi(active, starting, since 0.0100609s) 2024-04-16T05:31:35.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:35 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-16T05:31:35.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:35 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr metadata", "who": "smithi027.qgzypi", "id": "smithi027.qgzypi"}]: dispatch 2024-04-16T05:31:35.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:35 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T05:31:35.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:35 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T05:31:35.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:35 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T05:31:35.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:35 smithi027 ceph-mon[29206]: Manager daemon smithi027.qgzypi is now available 2024-04-16T05:31:35.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:35 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' 2024-04-16T05:31:36.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:36 smithi027 ceph-mon[29206]: Found migration_current of "None". Setting to last migration. 2024-04-16T05:31:36.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:36 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' 2024-04-16T05:31:36.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:36 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:31:36.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:36 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:31:36.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:36 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:31:36.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:36 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.qgzypi/mirror_snapshot_schedule"}]: dispatch 2024-04-16T05:31:36.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:36 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.qgzypi/trash_purge_schedule"}]: dispatch 2024-04-16T05:31:36.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:36 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' 2024-04-16T05:31:36.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:36 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:31:36.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:36 smithi027 ceph-mon[29206]: mgrmap e7: smithi027.qgzypi(active, since 1.01794s) 2024-04-16T05:31:36.766 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-16T05:31:36.767 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-16T05:31:36.767 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-16T05:31:36.767 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-16T05:31:36.767 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.168+0000 7f111f200700 1 Processor -- start 2024-04-16T05:31:36.767 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.168+0000 7f111f200700 1 -- start start 2024-04-16T05:31:36.767 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.169+0000 7f111f200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1118071290 0x7f1118107940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:36.774 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.169+0000 7f111f200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11180716a0 con 0x7f1118071290 2024-04-16T05:31:36.774 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.169+0000 7f1117e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1118071290 0x7f1118107940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:36.774 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.169+0000 7f1117e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1118071290 0x7f1118107940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59094/0 (socket says 172.21.15.27:59094) 2024-04-16T05:31:36.774 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.169+0000 7f1117e00700 1 -- 172.21.15.27:0/1467643118 learned_addr learned my addr 172.21.15.27:0/1467643118 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:36.774 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.169+0000 7f1117e00700 1 -- 172.21.15.27:0/1467643118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1118103330 con 0x7f1118071290 2024-04-16T05:31:36.774 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.170+0000 7f1117e00700 1 --2- 172.21.15.27:0/1467643118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1118071290 0x7f1118107940 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f11080098e0 tx=0x7f1108009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=578e2a9d8ee84a6f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:36.775 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.170+0000 7f1116a00700 1 -- 172.21.15.27:0/1467643118 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f110800b860 con 0x7f1118071290 2024-04-16T05:31:36.775 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.170+0000 7f1116a00700 1 -- 172.21.15.27:0/1467643118 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f110800be50 con 0x7f1118071290 2024-04-16T05:31:36.775 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.170+0000 7f1116a00700 1 -- 172.21.15.27:0/1467643118 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1108010380 con 0x7f1118071290 2024-04-16T05:31:36.775 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.171+0000 7f111f200700 1 -- 172.21.15.27:0/1467643118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1118071290 msgr2=0x7f1118107940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:36.775 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.171+0000 7f111f200700 1 --2- 172.21.15.27:0/1467643118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1118071290 0x7f1118107940 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f11080098e0 tx=0x7f1108009250 comp rx=0 tx=0).stop 2024-04-16T05:31:36.775 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.171+0000 7f111f200700 1 -- 172.21.15.27:0/1467643118 shutdown_connections 2024-04-16T05:31:36.775 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.171+0000 7f111f200700 1 --2- 172.21.15.27:0/1467643118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1118071290 0x7f1118107940 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:36.775 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.171+0000 7f111f200700 1 -- 172.21.15.27:0/1467643118 >> 172.21.15.27:0/1467643118 conn(0x7f11180fcfd0 msgr2=0x7f11180ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:36.775 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.171+0000 7f111f200700 1 -- 172.21.15.27:0/1467643118 shutdown_connections 2024-04-16T05:31:36.775 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.171+0000 7f111f200700 1 -- 172.21.15.27:0/1467643118 wait complete. 2024-04-16T05:31:36.775 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.172+0000 7f111f200700 1 Processor -- start 2024-04-16T05:31:36.775 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.172+0000 7f111f200700 1 -- start start 2024-04-16T05:31:36.775 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.173+0000 7f111f200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1118071290 0x7f11181137f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:36.775 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.173+0000 7f111f200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1118113d30 con 0x7f1118071290 2024-04-16T05:31:36.776 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.173+0000 7f1117e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1118071290 0x7f11181137f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:36.776 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.173+0000 7f1117e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1118071290 0x7f11181137f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59096/0 (socket says 172.21.15.27:59096) 2024-04-16T05:31:36.776 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.173+0000 7f1117e00700 1 -- 172.21.15.27:0/2621742499 learned_addr learned my addr 172.21.15.27:0/2621742499 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:36.776 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.173+0000 7f1117e00700 1 -- 172.21.15.27:0/2621742499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11080094f0 con 0x7f1118071290 2024-04-16T05:31:36.776 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.174+0000 7f1117e00700 1 --2- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1118071290 0x7f11181137f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1108006fd0 tx=0x7f1108009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:36.776 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.174+0000 7f110fe00700 1 -- 172.21.15.27:0/2621742499 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f110801a7d0 con 0x7f1118071290 2024-04-16T05:31:36.776 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.174+0000 7f110fe00700 1 -- 172.21.15.27:0/2621742499 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f110801adc0 con 0x7f1118071290 2024-04-16T05:31:36.776 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.174+0000 7f110fe00700 1 -- 172.21.15.27:0/2621742499 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1108019ea0 con 0x7f1118071290 2024-04-16T05:31:36.776 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.174+0000 7f111f200700 1 -- 172.21.15.27:0/2621742499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1118113f90 con 0x7f1118071290 2024-04-16T05:31:36.776 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.174+0000 7f111f200700 1 -- 172.21.15.27:0/2621742499 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1118114aa0 con 0x7f1118071290 2024-04-16T05:31:36.776 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.176+0000 7f110fe00700 1 -- 172.21.15.27:0/2621742499 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f110801a930 con 0x7f1118071290 2024-04-16T05:31:36.776 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.176+0000 7f110fe00700 1 --2- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f10fc036830 0x7f10fc038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:36.776 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.176+0000 7f110fe00700 1 -- 172.21.15.27:0/2621742499 --> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10fc0393f0 con 0x7f10fc036830 2024-04-16T05:31:36.776 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.176+0000 7f110fe00700 1 -- 172.21.15.27:0/2621742499 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1108049fe0 con 0x7f1118071290 2024-04-16T05:31:36.776 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.176+0000 7f1117400700 1 -- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f10fc036830 msgr2=0x7f10fc038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2173136120 2024-04-16T05:31:36.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.176+0000 7f1117400700 1 --2- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f10fc036830 0x7f10fc038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T05:31:36.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.377+0000 7f1117400700 1 -- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f10fc036830 msgr2=0x7f10fc038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2173136120 2024-04-16T05:31:36.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.377+0000 7f1117400700 1 --2- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f10fc036830 0x7f10fc038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T05:31:36.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.778+0000 7f1117400700 1 -- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f10fc036830 msgr2=0x7f10fc038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2173136120 2024-04-16T05:31:36.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:29.778+0000 7f1117400700 1 --2- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f10fc036830 0x7f10fc038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T05:31:36.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:30.579+0000 7f1117400700 1 -- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f10fc036830 msgr2=0x7f10fc038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2173136120 2024-04-16T05:31:36.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:30.579+0000 7f1117400700 1 --2- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f10fc036830 0x7f10fc038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T05:31:36.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:32.181+0000 7f1117400700 1 -- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f10fc036830 msgr2=0x7f10fc038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2173136120 2024-04-16T05:31:36.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:32.181+0000 7f1117400700 1 --2- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f10fc036830 0x7f10fc038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T05:31:36.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:35.000+0000 7f110fe00700 1 -- 172.21.15.27:0/2621742499 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mgrmap(e 6) v1 ==== 43143+0+0 (secure 0 0 0) 0x7f1108019730 con 0x7f1118071290 2024-04-16T05:31:36.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:35.000+0000 7f110fe00700 1 -- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f10fc036830 msgr2=0x7f10fc038ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T05:31:36.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:35.000+0000 7f110fe00700 1 --2- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f10fc036830 0x7f10fc038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:36.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:36.005+0000 7f110fe00700 1 -- 172.21.15.27:0/2621742499 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 7) v1 ==== 43270+0+0 (secure 0 0 0) 0x7f110804b770 con 0x7f1118071290 2024-04-16T05:31:36.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:36.006+0000 7f110fe00700 1 --2- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f10fc039fc0 0x7f10fc03c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:36.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:36.006+0000 7f110fe00700 1 -- 172.21.15.27:0/2621742499 --> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10fc0393f0 con 0x7f10fc039fc0 2024-04-16T05:31:36.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:36.007+0000 7f1117400700 1 --2- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f10fc039fc0 0x7f10fc03c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:36.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:36.012+0000 7f1117400700 1 --2- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f10fc039fc0 0x7f10fc03c3a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f11040039d0 tx=0x7f1104009350 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:36.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:36.014+0000 7f110fe00700 1 -- 172.21.15.27:0/2621742499 <== mgr.14120 v2:172.21.15.27:6800/2779081505 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f10fc0393f0 con 0x7f10fc039fc0 2024-04-16T05:31:36.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:36.022+0000 7f111f200700 1 -- 172.21.15.27:0/2621742499 --> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f111805f5f0 con 0x7f10fc039fc0 2024-04-16T05:31:36.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:36.023+0000 7f110fe00700 1 -- 172.21.15.27:0/2621742499 <== mgr.14120 v2:172.21.15.27:6800/2779081505 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f111805f5f0 con 0x7f10fc039fc0 2024-04-16T05:31:36.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:36.024+0000 7f111f200700 1 -- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f10fc039fc0 msgr2=0x7f10fc03c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:36.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:36.024+0000 7f111f200700 1 --2- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f10fc039fc0 0x7f10fc03c3a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f11040039d0 tx=0x7f1104009350 comp rx=0 tx=0).stop 2024-04-16T05:31:36.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:36.024+0000 7f111f200700 1 -- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1118071290 msgr2=0x7f11181137f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:36.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:36.024+0000 7f111f200700 1 --2- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1118071290 0x7f11181137f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1108006fd0 tx=0x7f1108009250 comp rx=0 tx=0).stop 2024-04-16T05:31:36.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:36.024+0000 7f111f200700 1 -- 172.21.15.27:0/2621742499 shutdown_connections 2024-04-16T05:31:36.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:36.024+0000 7f111f200700 1 --2- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f10fc039fc0 0x7f10fc03c3a0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:36.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:36.024+0000 7f111f200700 1 --2- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1118071290 0x7f11181137f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:36.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:36.024+0000 7f111f200700 1 --2- 172.21.15.27:0/2621742499 >> [v2:172.21.15.27:6800/2173136120,v1:172.21.15.27:6801/2173136120] conn(0x7f10fc036830 0x7f10fc038ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:36.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:36.024+0000 7f111f200700 1 -- 172.21.15.27:0/2621742499 >> 172.21.15.27:0/2621742499 conn(0x7f11180fcfd0 msgr2=0x7f11180fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:36.779 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:36.024+0000 7f111f200700 1 -- 172.21.15.27:0/2621742499 shutdown_connections 2024-04-16T05:31:36.779 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:36.025+0000 7f111f200700 1 -- 172.21.15.27:0/2621742499 wait complete. 2024-04-16T05:31:36.779 INFO:teuthology.orchestra.run.smithi027.stdout:mgr epoch 5 is available 2024-04-16T05:31:36.779 INFO:teuthology.orchestra.run.smithi027.stdout:Setting orchestrator backend to cephadm... 2024-04-16T05:31:37.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:37 smithi027 ceph-mon[29206]: [16/Apr/2024:05:31:35] ENGINE Bus STARTING 2024-04-16T05:31:37.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:37 smithi027 ceph-mon[29206]: [16/Apr/2024:05:31:35] ENGINE Serving on https://172.21.15.27:7150 2024-04-16T05:31:37.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:37 smithi027 ceph-mon[29206]: [16/Apr/2024:05:31:35] ENGINE Bus STARTED 2024-04-16T05:31:37.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:37 smithi027 ceph-mon[29206]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-16T05:31:37.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:37 smithi027 ceph-mon[29206]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-16T05:31:38.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:38 smithi027 ceph-mon[29206]: mgrmap e8: smithi027.qgzypi(active, since 2s) 2024-04-16T05:31:38.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:38 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' 2024-04-16T05:31:38.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:38 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:31:38.426 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.385+0000 7f6b0e400700 1 Processor -- start 2024-04-16T05:31:38.426 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.386+0000 7f6b0e400700 1 -- start start 2024-04-16T05:31:38.426 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.386+0000 7f6b0e400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b080712c0 0x7f6b08105520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:38.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.386+0000 7f6b0e400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b08105a60 con 0x7f6b080712c0 2024-04-16T05:31:38.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.386+0000 7f6b07400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b080712c0 0x7f6b08105520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:38.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.386+0000 7f6b07400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b080712c0 0x7f6b08105520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46846/0 (socket says 172.21.15.27:46846) 2024-04-16T05:31:38.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.386+0000 7f6b07400700 1 -- 172.21.15.27:0/2313463656 learned_addr learned my addr 172.21.15.27:0/2313463656 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:38.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.387+0000 7f6b07400700 1 -- 172.21.15.27:0/2313463656 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b08105ba0 con 0x7f6b080712c0 2024-04-16T05:31:38.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.387+0000 7f6b07400700 1 --2- 172.21.15.27:0/2313463656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b080712c0 0x7f6b08105520 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6af8009e50 tx=0x7f6af8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74aef333413fd20 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:38.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.387+0000 7f6b06a00700 1 -- 172.21.15.27:0/2313463656 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6af80092e0 con 0x7f6b080712c0 2024-04-16T05:31:38.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.388+0000 7f6b06a00700 1 -- 172.21.15.27:0/2313463656 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f6af8012430 con 0x7f6b080712c0 2024-04-16T05:31:38.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.388+0000 7f6b06a00700 1 -- 172.21.15.27:0/2313463656 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6af80075c0 con 0x7f6b080712c0 2024-04-16T05:31:38.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.388+0000 7f6b0e400700 1 -- 172.21.15.27:0/2313463656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b080712c0 msgr2=0x7f6b08105520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:38.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.388+0000 7f6b0e400700 1 --2- 172.21.15.27:0/2313463656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b080712c0 0x7f6b08105520 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6af8009e50 tx=0x7f6af8009840 comp rx=0 tx=0).stop 2024-04-16T05:31:38.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.388+0000 7f6b0e400700 1 -- 172.21.15.27:0/2313463656 shutdown_connections 2024-04-16T05:31:38.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.388+0000 7f6b0e400700 1 --2- 172.21.15.27:0/2313463656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b080712c0 0x7f6b08105520 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:38.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.388+0000 7f6b0e400700 1 -- 172.21.15.27:0/2313463656 >> 172.21.15.27:0/2313463656 conn(0x7f6b080fcfd0 msgr2=0x7f6b080ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:38.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.389+0000 7f6b0e400700 1 -- 172.21.15.27:0/2313463656 shutdown_connections 2024-04-16T05:31:38.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.389+0000 7f6b0e400700 1 -- 172.21.15.27:0/2313463656 wait complete. 2024-04-16T05:31:38.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.389+0000 7f6b0e400700 1 Processor -- start 2024-04-16T05:31:38.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.390+0000 7f6b0e400700 1 -- start start 2024-04-16T05:31:38.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.390+0000 7f6b0e400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b080712c0 0x7f6b08113890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:38.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.390+0000 7f6b0e400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b08113dd0 con 0x7f6b080712c0 2024-04-16T05:31:38.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.390+0000 7f6b07400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b080712c0 0x7f6b08113890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:38.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.390+0000 7f6b07400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b080712c0 0x7f6b08113890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46858/0 (socket says 172.21.15.27:46858) 2024-04-16T05:31:38.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.390+0000 7f6b07400700 1 -- 172.21.15.27:0/1957567827 learned_addr learned my addr 172.21.15.27:0/1957567827 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:38.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.391+0000 7f6b07400700 1 -- 172.21.15.27:0/1957567827 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af8009ae0 con 0x7f6b080712c0 2024-04-16T05:31:38.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.391+0000 7f6b07400700 1 --2- 172.21.15.27:0/1957567827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b080712c0 0x7f6b08113890 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6af8012fd0 tx=0x7f6af8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:38.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.391+0000 7f6affe00700 1 -- 172.21.15.27:0/1957567827 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6af800bc30 con 0x7f6b080712c0 2024-04-16T05:31:38.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.391+0000 7f6affe00700 1 -- 172.21.15.27:0/1957567827 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f6af800bd90 con 0x7f6b080712c0 2024-04-16T05:31:38.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.391+0000 7f6b0e400700 1 -- 172.21.15.27:0/1957567827 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b08113fd0 con 0x7f6b080712c0 2024-04-16T05:31:38.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.391+0000 7f6affe00700 1 -- 172.21.15.27:0/1957567827 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6af8025a90 con 0x7f6b080712c0 2024-04-16T05:31:38.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.391+0000 7f6b0e400700 1 -- 172.21.15.27:0/1957567827 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b08114b40 con 0x7f6b080712c0 2024-04-16T05:31:38.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.392+0000 7f6affe00700 1 -- 172.21.15.27:0/1957567827 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f6af8019050 con 0x7f6b080712c0 2024-04-16T05:31:38.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.393+0000 7f6b0e400700 1 -- 172.21.15.27:0/1957567827 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b0805f5f0 con 0x7f6b080712c0 2024-04-16T05:31:38.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.393+0000 7f6affe00700 1 --2- 172.21.15.27:0/1957567827 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f6aec0368a0 0x7f6aec038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:38.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.393+0000 7f6affe00700 1 -- 172.21.15.27:0/1957567827 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f6af804a220 con 0x7f6b080712c0 2024-04-16T05:31:38.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.393+0000 7f6afea00700 1 --2- 172.21.15.27:0/1957567827 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f6aec0368a0 0x7f6aec038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:38.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.394+0000 7f6afea00700 1 --2- 172.21.15.27:0/1957567827 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f6aec0368a0 0x7f6aec038d50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6af0009990 tx=0x7f6af0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:38.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.399+0000 7f6affe00700 1 -- 172.21.15.27:0/1957567827 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f6af802db40 con 0x7f6b080712c0 2024-04-16T05:31:38.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.632+0000 7f6b0e400700 1 -- 172.21.15.27:0/1957567827 --> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b080fe120 con 0x7f6aec0368a0 2024-04-16T05:31:38.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.648+0000 7f6affe00700 1 -- 172.21.15.27:0/1957567827 <== mgr.14120 v2:172.21.15.27:6800/2779081505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6b080fe120 con 0x7f6aec0368a0 2024-04-16T05:31:38.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.651+0000 7f6b0e400700 1 -- 172.21.15.27:0/1957567827 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f6aec0368a0 msgr2=0x7f6aec038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:38.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.651+0000 7f6b0e400700 1 --2- 172.21.15.27:0/1957567827 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f6aec0368a0 0x7f6aec038d50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6af0009990 tx=0x7f6af0006e40 comp rx=0 tx=0).stop 2024-04-16T05:31:38.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.651+0000 7f6b0e400700 1 -- 172.21.15.27:0/1957567827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b080712c0 msgr2=0x7f6b08113890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:38.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.651+0000 7f6b0e400700 1 --2- 172.21.15.27:0/1957567827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b080712c0 0x7f6b08113890 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6af8012fd0 tx=0x7f6af8009840 comp rx=0 tx=0).stop 2024-04-16T05:31:38.429 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.651+0000 7f6b0e400700 1 -- 172.21.15.27:0/1957567827 shutdown_connections 2024-04-16T05:31:38.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.651+0000 7f6b0e400700 1 --2- 172.21.15.27:0/1957567827 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f6aec0368a0 0x7f6aec038d50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:38.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.651+0000 7f6b0e400700 1 --2- 172.21.15.27:0/1957567827 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b080712c0 0x7f6b08113890 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:38.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.651+0000 7f6b0e400700 1 -- 172.21.15.27:0/1957567827 >> 172.21.15.27:0/1957567827 conn(0x7f6b080fcfd0 msgr2=0x7f6b080fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:38.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.652+0000 7f6b0e400700 1 -- 172.21.15.27:0/1957567827 shutdown_connections 2024-04-16T05:31:38.430 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:37.652+0000 7f6b0e400700 1 -- 172.21.15.27:0/1957567827 wait complete. 2024-04-16T05:31:39.275 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:39 smithi027 ceph-mon[29206]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:31:40.024 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-16T05:31:40.024 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.039+0000 7f8ee3400700 1 Processor -- start 2024-04-16T05:31:40.024 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.039+0000 7f8ee3400700 1 -- start start 2024-04-16T05:31:40.024 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.040+0000 7f8ee3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8edc102900 0x7f8edc102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:40.025 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.040+0000 7f8ee3400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc103250 con 0x7f8edc102900 2024-04-16T05:31:40.025 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.040+0000 7f8ee0a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8edc102900 0x7f8edc102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:40.025 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.040+0000 7f8ee0a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8edc102900 0x7f8edc102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46868/0 (socket says 172.21.15.27:46868) 2024-04-16T05:31:40.026 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.040+0000 7f8ee0a00700 1 -- 172.21.15.27:0/1865756508 learned_addr learned my addr 172.21.15.27:0/1865756508 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:40.026 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.041+0000 7f8ee0a00700 1 -- 172.21.15.27:0/1865756508 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8edc103390 con 0x7f8edc102900 2024-04-16T05:31:40.026 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.041+0000 7f8ee0a00700 1 --2- 172.21.15.27:0/1865756508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8edc102900 0x7f8edc102d10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00098e0 tx=0x7f8ed0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a659d7098f04bd12 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:40.027 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.041+0000 7f8edb400700 1 -- 172.21.15.27:0/1865756508 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ed000b860 con 0x7f8edc102900 2024-04-16T05:31:40.027 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.042+0000 7f8edb400700 1 -- 172.21.15.27:0/1865756508 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f8ed000be50 con 0x7f8edc102900 2024-04-16T05:31:40.027 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.042+0000 7f8edb400700 1 -- 172.21.15.27:0/1865756508 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ed0010380 con 0x7f8edc102900 2024-04-16T05:31:40.027 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.042+0000 7f8ee3400700 1 -- 172.21.15.27:0/1865756508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8edc102900 msgr2=0x7f8edc102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:40.027 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.042+0000 7f8ee3400700 1 --2- 172.21.15.27:0/1865756508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8edc102900 0x7f8edc102d10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00098e0 tx=0x7f8ed0009250 comp rx=0 tx=0).stop 2024-04-16T05:31:40.027 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.042+0000 7f8ee3400700 1 -- 172.21.15.27:0/1865756508 shutdown_connections 2024-04-16T05:31:40.027 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.042+0000 7f8ee3400700 1 --2- 172.21.15.27:0/1865756508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8edc102900 0x7f8edc102d10 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:40.027 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.042+0000 7f8ee3400700 1 -- 172.21.15.27:0/1865756508 >> 172.21.15.27:0/1865756508 conn(0x7f8edc0fe0a0 msgr2=0x7f8edc1004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:40.027 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.042+0000 7f8ee3400700 1 -- 172.21.15.27:0/1865756508 shutdown_connections 2024-04-16T05:31:40.027 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.042+0000 7f8ee3400700 1 -- 172.21.15.27:0/1865756508 wait complete. 2024-04-16T05:31:40.029 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.043+0000 7f8ee3400700 1 Processor -- start 2024-04-16T05:31:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.044+0000 7f8ee3400700 1 -- start start 2024-04-16T05:31:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.044+0000 7f8ee3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8edc102900 0x7f8edc111610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.044+0000 7f8ee3400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc111b50 con 0x7f8edc102900 2024-04-16T05:31:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.044+0000 7f8ee0a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8edc102900 0x7f8edc111610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.044+0000 7f8ee0a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8edc102900 0x7f8edc111610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46874/0 (socket says 172.21.15.27:46874) 2024-04-16T05:31:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.044+0000 7f8ee0a00700 1 -- 172.21.15.27:0/210462255 learned_addr learned my addr 172.21.15.27:0/210462255 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.045+0000 7f8ee0a00700 1 -- 172.21.15.27:0/210462255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed00094f0 con 0x7f8edc102900 2024-04-16T05:31:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.045+0000 7f8ee0a00700 1 --2- 172.21.15.27:0/210462255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8edc102900 0x7f8edc111610 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0000c00 tx=0x7f8ed0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.045+0000 7f8ed8c00700 1 -- 172.21.15.27:0/210462255 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ed001a8a0 con 0x7f8edc102900 2024-04-16T05:31:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.045+0000 7f8ed8c00700 1 -- 172.21.15.27:0/210462255 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f8ed001ae90 con 0x7f8edc102900 2024-04-16T05:31:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.045+0000 7f8ee3400700 1 -- 172.21.15.27:0/210462255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8edc111d50 con 0x7f8edc102900 2024-04-16T05:31:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.045+0000 7f8ed8c00700 1 -- 172.21.15.27:0/210462255 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ed0019e70 con 0x7f8edc102900 2024-04-16T05:31:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.045+0000 7f8ee3400700 1 -- 172.21.15.27:0/210462255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8edc1128c0 con 0x7f8edc102900 2024-04-16T05:31:40.030 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.046+0000 7f8ed8c00700 1 -- 172.21.15.27:0/210462255 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f8ed0018420 con 0x7f8edc102900 2024-04-16T05:31:40.031 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.047+0000 7f8ee3400700 1 -- 172.21.15.27:0/210462255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ebc0052e0 con 0x7f8edc102900 2024-04-16T05:31:40.031 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.047+0000 7f8ed8c00700 1 --2- 172.21.15.27:0/210462255 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f8ec40368a0 0x7f8ec4038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:40.031 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.047+0000 7f8ed8c00700 1 -- 172.21.15.27:0/210462255 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f8ed004a0d0 con 0x7f8edc102900 2024-04-16T05:31:40.031 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.047+0000 7f8edbe00700 1 --2- 172.21.15.27:0/210462255 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f8ec40368a0 0x7f8ec4038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:40.031 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.048+0000 7f8edbe00700 1 --2- 172.21.15.27:0/210462255 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f8ec40368a0 0x7f8ec4038d50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8009990 tx=0x7f8ec8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:40.031 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.053+0000 7f8ed8c00700 1 -- 172.21.15.27:0/210462255 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f8ed007c250 con 0x7f8edc102900 2024-04-16T05:31:40.031 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.275+0000 7f8ee3400700 1 -- 172.21.15.27:0/210462255 --> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ebc000bf0 con 0x7f8ec40368a0 2024-04-16T05:31:40.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.276+0000 7f8ed8c00700 1 -- 172.21.15.27:0/210462255 <== mgr.14120 v2:172.21.15.27:6800/2779081505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f8ebc000bf0 con 0x7f8ec40368a0 2024-04-16T05:31:40.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.280+0000 7f8ee3400700 1 -- 172.21.15.27:0/210462255 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f8ec40368a0 msgr2=0x7f8ec4038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:40.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.280+0000 7f8ee3400700 1 --2- 172.21.15.27:0/210462255 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f8ec40368a0 0x7f8ec4038d50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8009990 tx=0x7f8ec8006e40 comp rx=0 tx=0).stop 2024-04-16T05:31:40.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.280+0000 7f8ee3400700 1 -- 172.21.15.27:0/210462255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8edc102900 msgr2=0x7f8edc111610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:40.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.280+0000 7f8ee3400700 1 --2- 172.21.15.27:0/210462255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8edc102900 0x7f8edc111610 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0000c00 tx=0x7f8ed0009250 comp rx=0 tx=0).stop 2024-04-16T05:31:40.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.281+0000 7f8ee3400700 1 -- 172.21.15.27:0/210462255 shutdown_connections 2024-04-16T05:31:40.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.281+0000 7f8ee3400700 1 --2- 172.21.15.27:0/210462255 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f8ec40368a0 0x7f8ec4038d50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:40.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.281+0000 7f8ee3400700 1 --2- 172.21.15.27:0/210462255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8edc102900 0x7f8edc111610 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:40.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.281+0000 7f8ee3400700 1 -- 172.21.15.27:0/210462255 >> 172.21.15.27:0/210462255 conn(0x7f8edc0fe0a0 msgr2=0x7f8edc0feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:40.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.281+0000 7f8ee3400700 1 -- 172.21.15.27:0/210462255 shutdown_connections 2024-04-16T05:31:40.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:39.282+0000 7f8ee3400700 1 -- 172.21.15.27:0/210462255 wait complete. 2024-04-16T05:31:40.033 INFO:teuthology.orchestra.run.smithi027.stdout:Generating ssh key... 2024-04-16T05:31:40.629 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:40 smithi027 ceph-mon[29206]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:31:41.798 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:41 smithi027 ceph-mon[29206]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:31:41.798 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:41 smithi027 ceph-mon[29206]: Generating ssh key... 2024-04-16T05:31:42.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.685+0000 7fb7e9a00700 1 Processor -- start 2024-04-16T05:31:42.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.685+0000 7fb7e9a00700 1 -- start start 2024-04-16T05:31:42.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.686+0000 7fb7e9a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7e407ca40 0x7fb7e407ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:42.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.686+0000 7fb7e9a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e407d420 con 0x7fb7e407ca40 2024-04-16T05:31:42.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.686+0000 7fb7e2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7e407ca40 0x7fb7e407ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:42.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.686+0000 7fb7e2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7e407ca40 0x7fb7e407ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46886/0 (socket says 172.21.15.27:46886) 2024-04-16T05:31:42.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.686+0000 7fb7e2a00700 1 -- 172.21.15.27:0/4287395948 learned_addr learned my addr 172.21.15.27:0/4287395948 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:42.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.687+0000 7fb7e2a00700 1 -- 172.21.15.27:0/4287395948 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e407d560 con 0x7fb7e407ca40 2024-04-16T05:31:42.530 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.687+0000 7fb7e2a00700 1 --2- 172.21.15.27:0/4287395948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7e407ca40 0x7fb7e407ce50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb7d40098e0 tx=0x7fb7d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75f50119f86460c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:42.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.688+0000 7fb7e1600700 1 -- 172.21.15.27:0/4287395948 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7d400b860 con 0x7fb7e407ca40 2024-04-16T05:31:42.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.688+0000 7fb7e1600700 1 -- 172.21.15.27:0/4287395948 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fb7d400be50 con 0x7fb7e407ca40 2024-04-16T05:31:42.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.688+0000 7fb7e1600700 1 -- 172.21.15.27:0/4287395948 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7d4010380 con 0x7fb7e407ca40 2024-04-16T05:31:42.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.688+0000 7fb7e9a00700 1 -- 172.21.15.27:0/4287395948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7e407ca40 msgr2=0x7fb7e407ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:42.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.688+0000 7fb7e9a00700 1 --2- 172.21.15.27:0/4287395948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7e407ca40 0x7fb7e407ce50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb7d40098e0 tx=0x7fb7d4009250 comp rx=0 tx=0).stop 2024-04-16T05:31:42.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.689+0000 7fb7e9a00700 1 -- 172.21.15.27:0/4287395948 shutdown_connections 2024-04-16T05:31:42.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.689+0000 7fb7e9a00700 1 --2- 172.21.15.27:0/4287395948 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7e407ca40 0x7fb7e407ce50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:42.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.689+0000 7fb7e9a00700 1 -- 172.21.15.27:0/4287395948 >> 172.21.15.27:0/4287395948 conn(0x7fb7e4078350 msgr2=0x7fb7e407a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:42.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.689+0000 7fb7e9a00700 1 -- 172.21.15.27:0/4287395948 shutdown_connections 2024-04-16T05:31:42.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.689+0000 7fb7e9a00700 1 -- 172.21.15.27:0/4287395948 wait complete. 2024-04-16T05:31:42.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.690+0000 7fb7e9a00700 1 Processor -- start 2024-04-16T05:31:42.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.690+0000 7fb7e9a00700 1 -- start start 2024-04-16T05:31:42.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.690+0000 7fb7e9a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7e407ca40 0x7fb7e4117be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:42.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.690+0000 7fb7e9a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e4118120 con 0x7fb7e407ca40 2024-04-16T05:31:42.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.690+0000 7fb7e2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7e407ca40 0x7fb7e4117be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:42.531 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.690+0000 7fb7e2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7e407ca40 0x7fb7e4117be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46896/0 (socket says 172.21.15.27:46896) 2024-04-16T05:31:42.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.690+0000 7fb7e2a00700 1 -- 172.21.15.27:0/1968340050 learned_addr learned my addr 172.21.15.27:0/1968340050 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:42.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.691+0000 7fb7e2a00700 1 -- 172.21.15.27:0/1968340050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d40094f0 con 0x7fb7e407ca40 2024-04-16T05:31:42.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.691+0000 7fb7e2a00700 1 --2- 172.21.15.27:0/1968340050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7e407ca40 0x7fb7e4117be0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4000c00 tx=0x7fb7d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:42.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.691+0000 7fb7daa00700 1 -- 172.21.15.27:0/1968340050 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7d401a8a0 con 0x7fb7e407ca40 2024-04-16T05:31:42.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.691+0000 7fb7daa00700 1 -- 172.21.15.27:0/1968340050 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fb7d401ae90 con 0x7fb7e407ca40 2024-04-16T05:31:42.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.691+0000 7fb7daa00700 1 -- 172.21.15.27:0/1968340050 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7d4019e70 con 0x7fb7e407ca40 2024-04-16T05:31:42.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.691+0000 7fb7e9a00700 1 -- 172.21.15.27:0/1968340050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e4118320 con 0x7fb7e407ca40 2024-04-16T05:31:42.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.692+0000 7fb7e9a00700 1 -- 172.21.15.27:0/1968340050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e4118e90 con 0x7fb7e407ca40 2024-04-16T05:31:42.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.693+0000 7fb7daa00700 1 -- 172.21.15.27:0/1968340050 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fb7d4019730 con 0x7fb7e407ca40 2024-04-16T05:31:42.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.693+0000 7fb7e9a00700 1 -- 172.21.15.27:0/1968340050 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7e405f5f0 con 0x7fb7e407ca40 2024-04-16T05:31:42.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.693+0000 7fb7daa00700 1 --2- 172.21.15.27:0/1968340050 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fb7c80368a0 0x7fb7c8038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:42.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.693+0000 7fb7daa00700 1 -- 172.21.15.27:0/1968340050 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb7d404a130 con 0x7fb7e407ca40 2024-04-16T05:31:42.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.693+0000 7fb7e2000700 1 --2- 172.21.15.27:0/1968340050 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fb7c80368a0 0x7fb7c8038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:42.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.694+0000 7fb7e2000700 1 --2- 172.21.15.27:0/1968340050 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fb7c80368a0 0x7fb7c8038d50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0009990 tx=0x7fb7d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:42.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.698+0000 7fb7daa00700 1 -- 172.21.15.27:0/1968340050 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb7d4021b10 con 0x7fb7e407ca40 2024-04-16T05:31:42.532 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:40.913+0000 7fb7e9a00700 1 -- 172.21.15.27:0/1968340050 --> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7e407a4d0 con 0x7fb7c80368a0 2024-04-16T05:31:42.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:41.824+0000 7fb7daa00700 1 -- 172.21.15.27:0/1968340050 <== mgr.14120 v2:172.21.15.27:6800/2779081505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb7e407a4d0 con 0x7fb7c80368a0 2024-04-16T05:31:42.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:41.827+0000 7fb7e9a00700 1 -- 172.21.15.27:0/1968340050 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fb7c80368a0 msgr2=0x7fb7c8038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:42.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:41.827+0000 7fb7e9a00700 1 --2- 172.21.15.27:0/1968340050 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fb7c80368a0 0x7fb7c8038d50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0009990 tx=0x7fb7d0006e40 comp rx=0 tx=0).stop 2024-04-16T05:31:42.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:41.827+0000 7fb7e9a00700 1 -- 172.21.15.27:0/1968340050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7e407ca40 msgr2=0x7fb7e4117be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:42.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:41.827+0000 7fb7e9a00700 1 --2- 172.21.15.27:0/1968340050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7e407ca40 0x7fb7e4117be0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4000c00 tx=0x7fb7d4009250 comp rx=0 tx=0).stop 2024-04-16T05:31:42.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:41.827+0000 7fb7e9a00700 1 -- 172.21.15.27:0/1968340050 shutdown_connections 2024-04-16T05:31:42.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:41.827+0000 7fb7e9a00700 1 --2- 172.21.15.27:0/1968340050 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fb7c80368a0 0x7fb7c8038d50 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:42.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:41.827+0000 7fb7e9a00700 1 --2- 172.21.15.27:0/1968340050 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7e407ca40 0x7fb7e4117be0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:42.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:41.827+0000 7fb7e9a00700 1 -- 172.21.15.27:0/1968340050 >> 172.21.15.27:0/1968340050 conn(0x7fb7e4078350 msgr2=0x7fb7e4079dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:42.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:41.827+0000 7fb7e9a00700 1 -- 172.21.15.27:0/1968340050 shutdown_connections 2024-04-16T05:31:42.533 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:41.827+0000 7fb7e9a00700 1 -- 172.21.15.27:0/1968340050 wait complete. 2024-04-16T05:31:43.076 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:42 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' 2024-04-16T05:31:43.076 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:42 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' 2024-04-16T05:31:44.053 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIwdQH4aotIZd9sP60hGpybsp0U0UEe/hy0wcbh+eT0SDhe2RhSv2RbpvBReQDxqlBo72AYaVPV2fpVezYRLbid0tRka6jbYtDITICfEi3dsM0hx+wns2fFahNEAGgcOYnO5oWl4J2GAzXtvU8NVXCYE/H4MRWjfSi3I+j/hV/CVOb6HeQ/q4skiJ0CfMmo+KUd7dgs3KAparIXq2HEzr+EOaUCXG34rzn7Yl7K8UhKCbBOBZw2ut/KTlK3+7rjh4q/3vTpVC1kr8AjeGHQIRovJ+TD2MOQXKO2T2Eo3qE5vTeAqhIbL67n7wzGk7w4EnbQ6zWJ1rvBPZ7hUvv34AAi4Wj0swFO0xCoJfh7izLrew5OS43OwvQCxB+GjprIGpSxlEqM61QDxJOFe2nLrCFWSurZrhb4WwsmqcXIc8UG2HokTlo1tJVMu7qz0iAs1t5WQ7JWLBlqYnqcYWQwux9wyv0/6r/FJVdRtW5ps+idKxNrOXhcbGsT/h5fvRCqYk= ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:31:44.054 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.126+0000 7f2ea4a00700 1 Processor -- start 2024-04-16T05:31:44.054 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.126+0000 7f2ea4a00700 1 -- start start 2024-04-16T05:31:44.054 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.127+0000 7f2ea4a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ea007ca40 0x7f2ea007ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:44.054 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.127+0000 7f2ea4a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea007d420 con 0x7f2ea007ca40 2024-04-16T05:31:44.054 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.127+0000 7f2e9de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ea007ca40 0x7f2ea007ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:44.054 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.127+0000 7f2e9de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ea007ca40 0x7f2ea007ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46902/0 (socket says 172.21.15.27:46902) 2024-04-16T05:31:44.054 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.127+0000 7f2e9de00700 1 -- 172.21.15.27:0/1168156379 learned_addr learned my addr 172.21.15.27:0/1168156379 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:44.054 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.128+0000 7f2e9de00700 1 -- 172.21.15.27:0/1168156379 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea007d560 con 0x7f2ea007ca40 2024-04-16T05:31:44.054 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.128+0000 7f2e9de00700 1 --2- 172.21.15.27:0/1168156379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ea007ca40 0x7f2ea007ce50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2e900098e0 tx=0x7f2e90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61295090ba2ce72d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:44.054 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.128+0000 7f2e9ca00700 1 -- 172.21.15.27:0/1168156379 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e9000b860 con 0x7f2ea007ca40 2024-04-16T05:31:44.058 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.128+0000 7f2e9ca00700 1 -- 172.21.15.27:0/1168156379 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f2e9000be50 con 0x7f2ea007ca40 2024-04-16T05:31:44.058 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.128+0000 7f2e9ca00700 1 -- 172.21.15.27:0/1168156379 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e90010380 con 0x7f2ea007ca40 2024-04-16T05:31:44.059 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.129+0000 7f2ea4a00700 1 -- 172.21.15.27:0/1168156379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ea007ca40 msgr2=0x7f2ea007ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:44.059 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.129+0000 7f2ea4a00700 1 --2- 172.21.15.27:0/1168156379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ea007ca40 0x7f2ea007ce50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2e900098e0 tx=0x7f2e90009250 comp rx=0 tx=0).stop 2024-04-16T05:31:44.059 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.129+0000 7f2ea4a00700 1 -- 172.21.15.27:0/1168156379 shutdown_connections 2024-04-16T05:31:44.059 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.129+0000 7f2ea4a00700 1 --2- 172.21.15.27:0/1168156379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ea007ca40 0x7f2ea007ce50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:44.059 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.129+0000 7f2ea4a00700 1 -- 172.21.15.27:0/1168156379 >> 172.21.15.27:0/1168156379 conn(0x7f2ea0078350 msgr2=0x7f2ea007a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:44.059 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.129+0000 7f2ea4a00700 1 -- 172.21.15.27:0/1168156379 shutdown_connections 2024-04-16T05:31:44.059 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.129+0000 7f2ea4a00700 1 -- 172.21.15.27:0/1168156379 wait complete. 2024-04-16T05:31:44.059 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.130+0000 7f2ea4a00700 1 Processor -- start 2024-04-16T05:31:44.059 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.130+0000 7f2ea4a00700 1 -- start start 2024-04-16T05:31:44.059 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.131+0000 7f2ea4a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ea007ca40 0x7f2ea0117be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:44.059 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.131+0000 7f2ea4a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea0118120 con 0x7f2ea007ca40 2024-04-16T05:31:44.059 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.131+0000 7f2e9de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ea007ca40 0x7f2ea0117be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:44.060 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.131+0000 7f2e9de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ea007ca40 0x7f2ea0117be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46914/0 (socket says 172.21.15.27:46914) 2024-04-16T05:31:44.060 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.131+0000 7f2e9de00700 1 -- 172.21.15.27:0/3424906603 learned_addr learned my addr 172.21.15.27:0/3424906603 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:44.060 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.131+0000 7f2e9de00700 1 -- 172.21.15.27:0/3424906603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e900094f0 con 0x7f2ea007ca40 2024-04-16T05:31:44.060 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.132+0000 7f2e9de00700 1 --2- 172.21.15.27:0/3424906603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ea007ca40 0x7f2ea0117be0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2e90006fd0 tx=0x7f2e90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:44.060 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.132+0000 7f2e96000700 1 -- 172.21.15.27:0/3424906603 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e90010800 con 0x7f2ea007ca40 2024-04-16T05:31:44.060 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.132+0000 7f2e96000700 1 -- 172.21.15.27:0/3424906603 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f2e90010df0 con 0x7f2ea007ca40 2024-04-16T05:31:44.060 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.132+0000 7f2ea4a00700 1 -- 172.21.15.27:0/3424906603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea0118320 con 0x7f2ea007ca40 2024-04-16T05:31:44.060 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.132+0000 7f2e96000700 1 -- 172.21.15.27:0/3424906603 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e9001ad80 con 0x7f2ea007ca40 2024-04-16T05:31:44.060 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.132+0000 7f2ea4a00700 1 -- 172.21.15.27:0/3424906603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea0118e90 con 0x7f2ea007ca40 2024-04-16T05:31:44.060 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.133+0000 7f2ea4a00700 1 -- 172.21.15.27:0/3424906603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ea005f5f0 con 0x7f2ea007ca40 2024-04-16T05:31:44.060 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.133+0000 7f2e96000700 1 -- 172.21.15.27:0/3424906603 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f2e9001a660 con 0x7f2ea007ca40 2024-04-16T05:31:44.060 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.134+0000 7f2e96000700 1 --2- 172.21.15.27:0/3424906603 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f2e840368a0 0x7f2e84038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:44.060 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.134+0000 7f2e96000700 1 -- 172.21.15.27:0/3424906603 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f2e9004a060 con 0x7f2ea007ca40 2024-04-16T05:31:44.060 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.134+0000 7f2e9d400700 1 --2- 172.21.15.27:0/3424906603 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f2e840368a0 0x7f2e84038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:44.060 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.135+0000 7f2e9d400700 1 --2- 172.21.15.27:0/3424906603 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f2e840368a0 0x7f2e84038d50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c009990 tx=0x7f2e8c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:44.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.139+0000 7f2e96000700 1 -- 172.21.15.27:0/3424906603 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f2e900228d0 con 0x7f2ea007ca40 2024-04-16T05:31:44.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.356+0000 7f2ea4a00700 1 -- 172.21.15.27:0/3424906603 --> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ea007a4d0 con 0x7f2e840368a0 2024-04-16T05:31:44.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.358+0000 7f2e96000700 1 -- 172.21.15.27:0/3424906603 <== mgr.14120 v2:172.21.15.27:6800/2779081505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f2ea007a4d0 con 0x7f2e840368a0 2024-04-16T05:31:44.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.360+0000 7f2ea4a00700 1 -- 172.21.15.27:0/3424906603 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f2e840368a0 msgr2=0x7f2e84038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:44.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.360+0000 7f2ea4a00700 1 --2- 172.21.15.27:0/3424906603 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f2e840368a0 0x7f2e84038d50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c009990 tx=0x7f2e8c006e40 comp rx=0 tx=0).stop 2024-04-16T05:31:44.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.360+0000 7f2ea4a00700 1 -- 172.21.15.27:0/3424906603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ea007ca40 msgr2=0x7f2ea0117be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:44.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.360+0000 7f2ea4a00700 1 --2- 172.21.15.27:0/3424906603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ea007ca40 0x7f2ea0117be0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2e90006fd0 tx=0x7f2e90009250 comp rx=0 tx=0).stop 2024-04-16T05:31:44.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.361+0000 7f2ea4a00700 1 -- 172.21.15.27:0/3424906603 shutdown_connections 2024-04-16T05:31:44.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.361+0000 7f2ea4a00700 1 --2- 172.21.15.27:0/3424906603 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f2e840368a0 0x7f2e84038d50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:44.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.361+0000 7f2ea4a00700 1 --2- 172.21.15.27:0/3424906603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2ea007ca40 0x7f2ea0117be0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:44.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.361+0000 7f2ea4a00700 1 -- 172.21.15.27:0/3424906603 >> 172.21.15.27:0/3424906603 conn(0x7f2ea0078350 msgr2=0x7f2ea0079dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:44.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.361+0000 7f2ea4a00700 1 -- 172.21.15.27:0/3424906603 shutdown_connections 2024-04-16T05:31:44.061 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:43.362+0000 7f2ea4a00700 1 -- 172.21.15.27:0/3424906603 wait complete. 2024-04-16T05:31:44.062 INFO:teuthology.orchestra.run.smithi027.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-16T05:31:44.062 INFO:teuthology.orchestra.run.smithi027.stdout:Adding key to root@localhost authorized_keys... 2024-04-16T05:31:44.062 INFO:teuthology.orchestra.run.smithi027.stdout:Adding host smithi027... 2024-04-16T05:31:44.625 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:44 smithi027 ceph-mon[29206]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:31:45.468 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:45 smithi027 ceph-mon[29206]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi027", "addr": "172.21.15.27", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:31:47.718 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:47 smithi027 ceph-mon[29206]: Deploying cephadm binary to smithi027 2024-04-16T05:31:50.208 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Added host 'smithi027' with addr '172.21.15.27' 2024-04-16T05:31:50.208 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.697+0000 7f199ac00700 1 Processor -- start 2024-04-16T05:31:50.209 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.698+0000 7f199ac00700 1 -- start start 2024-04-16T05:31:50.209 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.698+0000 7f199ac00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f199407ca40 0x7f199407ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:50.209 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.698+0000 7f199ac00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199407d420 con 0x7f199407ca40 2024-04-16T05:31:50.209 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.699+0000 7f1993e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f199407ca40 0x7f199407ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:50.209 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.699+0000 7f1993e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f199407ca40 0x7f199407ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54520/0 (socket says 172.21.15.27:54520) 2024-04-16T05:31:50.209 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.699+0000 7f1993e00700 1 -- 172.21.15.27:0/3302252186 learned_addr learned my addr 172.21.15.27:0/3302252186 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:50.209 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.699+0000 7f1993e00700 1 -- 172.21.15.27:0/3302252186 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199407d560 con 0x7f199407ca40 2024-04-16T05:31:50.209 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.699+0000 7f1993e00700 1 --2- 172.21.15.27:0/3302252186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f199407ca40 0x7f199407ce50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f19840098e0 tx=0x7f1984009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e8e7e64bfe0a661 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:50.209 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.700+0000 7f1992a00700 1 -- 172.21.15.27:0/3302252186 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f198400b860 con 0x7f199407ca40 2024-04-16T05:31:50.209 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.700+0000 7f1992a00700 1 -- 172.21.15.27:0/3302252186 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f198400be50 con 0x7f199407ca40 2024-04-16T05:31:50.209 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.700+0000 7f1992a00700 1 -- 172.21.15.27:0/3302252186 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1984010380 con 0x7f199407ca40 2024-04-16T05:31:50.209 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.700+0000 7f199ac00700 1 -- 172.21.15.27:0/3302252186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f199407ca40 msgr2=0x7f199407ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:50.209 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.700+0000 7f199ac00700 1 --2- 172.21.15.27:0/3302252186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f199407ca40 0x7f199407ce50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f19840098e0 tx=0x7f1984009250 comp rx=0 tx=0).stop 2024-04-16T05:31:50.210 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.701+0000 7f199ac00700 1 -- 172.21.15.27:0/3302252186 shutdown_connections 2024-04-16T05:31:50.210 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.701+0000 7f199ac00700 1 --2- 172.21.15.27:0/3302252186 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f199407ca40 0x7f199407ce50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:50.210 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.701+0000 7f199ac00700 1 -- 172.21.15.27:0/3302252186 >> 172.21.15.27:0/3302252186 conn(0x7f1994078350 msgr2=0x7f199407a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:50.210 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.701+0000 7f199ac00700 1 -- 172.21.15.27:0/3302252186 shutdown_connections 2024-04-16T05:31:50.210 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.701+0000 7f199ac00700 1 -- 172.21.15.27:0/3302252186 wait complete. 2024-04-16T05:31:50.210 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.701+0000 7f199ac00700 1 Processor -- start 2024-04-16T05:31:50.210 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.702+0000 7f199ac00700 1 -- start start 2024-04-16T05:31:50.210 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.702+0000 7f199ac00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f199407ca40 0x7f1994117be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:50.210 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.702+0000 7f199ac00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1994118120 con 0x7f199407ca40 2024-04-16T05:31:50.210 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.702+0000 7f1993e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f199407ca40 0x7f1994117be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:50.210 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.702+0000 7f1993e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f199407ca40 0x7f1994117be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54526/0 (socket says 172.21.15.27:54526) 2024-04-16T05:31:50.210 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.702+0000 7f1993e00700 1 -- 172.21.15.27:0/3136554815 learned_addr learned my addr 172.21.15.27:0/3136554815 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:50.210 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.703+0000 7f1993e00700 1 -- 172.21.15.27:0/3136554815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19840094f0 con 0x7f199407ca40 2024-04-16T05:31:50.210 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.703+0000 7f1993e00700 1 --2- 172.21.15.27:0/3136554815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f199407ca40 0x7f1994117be0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1984000c00 tx=0x7f1984009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:50.210 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.703+0000 7f198be00700 1 -- 172.21.15.27:0/3136554815 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f198401a8a0 con 0x7f199407ca40 2024-04-16T05:31:50.211 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.703+0000 7f198be00700 1 -- 172.21.15.27:0/3136554815 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7f198401ae90 con 0x7f199407ca40 2024-04-16T05:31:50.211 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.703+0000 7f199ac00700 1 -- 172.21.15.27:0/3136554815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1994118320 con 0x7f199407ca40 2024-04-16T05:31:50.211 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.703+0000 7f198be00700 1 -- 172.21.15.27:0/3136554815 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1984019e70 con 0x7f199407ca40 2024-04-16T05:31:50.211 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.703+0000 7f199ac00700 1 -- 172.21.15.27:0/3136554815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1994118e90 con 0x7f199407ca40 2024-04-16T05:31:50.211 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.705+0000 7f198be00700 1 -- 172.21.15.27:0/3136554815 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f1984019730 con 0x7f199407ca40 2024-04-16T05:31:50.211 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.705+0000 7f199ac00700 1 -- 172.21.15.27:0/3136554815 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f199405f5f0 con 0x7f199407ca40 2024-04-16T05:31:50.211 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.705+0000 7f198be00700 1 --2- 172.21.15.27:0/3136554815 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f19780368a0 0x7f1978038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:50.211 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.705+0000 7f198be00700 1 -- 172.21.15.27:0/3136554815 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f198404a130 con 0x7f199407ca40 2024-04-16T05:31:50.211 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.705+0000 7f1993400700 1 --2- 172.21.15.27:0/3136554815 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f19780368a0 0x7f1978038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:50.211 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.706+0000 7f1993400700 1 --2- 172.21.15.27:0/3136554815 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f19780368a0 0x7f1978038d50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1980009990 tx=0x7f1980006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:50.211 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.711+0000 7f198be00700 1 -- 172.21.15.27:0/3136554815 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f1984021b10 con 0x7f199407ca40 2024-04-16T05:31:50.211 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:44.933+0000 7f199ac00700 1 -- 172.21.15.27:0/3136554815 --> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi027", "addr": "172.21.15.27", "target": ["mon-mgr", ""]}) v1 -- 0x7f199407a4b0 con 0x7f19780368a0 2024-04-16T05:31:50.211 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:49.447+0000 7f198be00700 1 -- 172.21.15.27:0/3136554815 <== mgr.14120 v2:172.21.15.27:6800/2779081505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f199407a4b0 con 0x7f19780368a0 2024-04-16T05:31:50.211 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:49.451+0000 7f199ac00700 1 -- 172.21.15.27:0/3136554815 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f19780368a0 msgr2=0x7f1978038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:50.211 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:49.451+0000 7f199ac00700 1 --2- 172.21.15.27:0/3136554815 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f19780368a0 0x7f1978038d50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1980009990 tx=0x7f1980006e40 comp rx=0 tx=0).stop 2024-04-16T05:31:50.212 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:49.451+0000 7f199ac00700 1 -- 172.21.15.27:0/3136554815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f199407ca40 msgr2=0x7f1994117be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:50.212 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:49.451+0000 7f199ac00700 1 --2- 172.21.15.27:0/3136554815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f199407ca40 0x7f1994117be0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1984000c00 tx=0x7f1984009250 comp rx=0 tx=0).stop 2024-04-16T05:31:50.212 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:49.451+0000 7f199ac00700 1 -- 172.21.15.27:0/3136554815 shutdown_connections 2024-04-16T05:31:50.212 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:49.451+0000 7f199ac00700 1 --2- 172.21.15.27:0/3136554815 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f19780368a0 0x7f1978038d50 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:50.212 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:49.451+0000 7f199ac00700 1 --2- 172.21.15.27:0/3136554815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f199407ca40 0x7f1994117be0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:50.212 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:49.451+0000 7f199ac00700 1 -- 172.21.15.27:0/3136554815 >> 172.21.15.27:0/3136554815 conn(0x7f1994078350 msgr2=0x7f1994079da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:50.212 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:49.452+0000 7f199ac00700 1 -- 172.21.15.27:0/3136554815 shutdown_connections 2024-04-16T05:31:50.212 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:49.452+0000 7f199ac00700 1 -- 172.21.15.27:0/3136554815 wait complete. 2024-04-16T05:31:50.212 INFO:teuthology.orchestra.run.smithi027.stdout:Deploying mon service with default placement... 2024-04-16T05:31:50.496 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:50 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' 2024-04-16T05:31:50.496 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:50 smithi027 ceph-mon[29206]: Added host smithi027 2024-04-16T05:31:50.496 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:50 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:31:52.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:52 smithi027 ceph-mon[29206]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:31:52.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:52 smithi027 ceph-mon[29206]: Saving service mon spec with placement count:5 2024-04-16T05:31:52.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:52 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' 2024-04-16T05:31:52.652 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-16T05:31:52.653 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.818+0000 7fa15e200700 1 Processor -- start 2024-04-16T05:31:52.653 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.819+0000 7fa15e200700 1 -- start start 2024-04-16T05:31:52.653 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.819+0000 7fa15e200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa15807ca40 0x7fa15807ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:52.653 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.819+0000 7fa15e200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15807d420 con 0x7fa15807ca40 2024-04-16T05:31:52.654 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.820+0000 7fa157400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa15807ca40 0x7fa15807ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:52.654 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.820+0000 7fa157400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa15807ca40 0x7fa15807ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54538/0 (socket says 172.21.15.27:54538) 2024-04-16T05:31:52.654 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.820+0000 7fa157400700 1 -- 172.21.15.27:0/1250680022 learned_addr learned my addr 172.21.15.27:0/1250680022 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:52.654 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.820+0000 7fa157400700 1 -- 172.21.15.27:0/1250680022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15807d560 con 0x7fa15807ca40 2024-04-16T05:31:52.654 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.821+0000 7fa157400700 1 --2- 172.21.15.27:0/1250680022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa15807ca40 0x7fa15807ce50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa1480098e0 tx=0x7fa148009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=540a6e862193e189 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:52.654 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.821+0000 7fa156000700 1 -- 172.21.15.27:0/1250680022 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa14800b860 con 0x7fa15807ca40 2024-04-16T05:31:52.654 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.821+0000 7fa156000700 1 -- 172.21.15.27:0/1250680022 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fa14800be50 con 0x7fa15807ca40 2024-04-16T05:31:52.655 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.821+0000 7fa156000700 1 -- 172.21.15.27:0/1250680022 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa148010380 con 0x7fa15807ca40 2024-04-16T05:31:52.656 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.822+0000 7fa15e200700 1 -- 172.21.15.27:0/1250680022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa15807ca40 msgr2=0x7fa15807ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:52.656 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.822+0000 7fa15e200700 1 --2- 172.21.15.27:0/1250680022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa15807ca40 0x7fa15807ce50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa1480098e0 tx=0x7fa148009250 comp rx=0 tx=0).stop 2024-04-16T05:31:52.656 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.822+0000 7fa15e200700 1 -- 172.21.15.27:0/1250680022 shutdown_connections 2024-04-16T05:31:52.656 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.822+0000 7fa15e200700 1 --2- 172.21.15.27:0/1250680022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa15807ca40 0x7fa15807ce50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:52.656 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.822+0000 7fa15e200700 1 -- 172.21.15.27:0/1250680022 >> 172.21.15.27:0/1250680022 conn(0x7fa158078350 msgr2=0x7fa15807a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:52.656 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.822+0000 7fa15e200700 1 -- 172.21.15.27:0/1250680022 shutdown_connections 2024-04-16T05:31:52.656 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.822+0000 7fa15e200700 1 -- 172.21.15.27:0/1250680022 wait complete. 2024-04-16T05:31:52.656 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.823+0000 7fa15e200700 1 Processor -- start 2024-04-16T05:31:52.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.823+0000 7fa15e200700 1 -- start start 2024-04-16T05:31:52.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.824+0000 7fa15e200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa15807ca40 0x7fa158117be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:52.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.824+0000 7fa15e200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa158118120 con 0x7fa15807ca40 2024-04-16T05:31:52.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.824+0000 7fa157400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa15807ca40 0x7fa158117be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:52.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.824+0000 7fa157400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa15807ca40 0x7fa158117be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54544/0 (socket says 172.21.15.27:54544) 2024-04-16T05:31:52.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.824+0000 7fa157400700 1 -- 172.21.15.27:0/350752934 learned_addr learned my addr 172.21.15.27:0/350752934 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:52.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.824+0000 7fa157400700 1 -- 172.21.15.27:0/350752934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1480094f0 con 0x7fa15807ca40 2024-04-16T05:31:52.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.825+0000 7fa157400700 1 --2- 172.21.15.27:0/350752934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa15807ca40 0x7fa158117be0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa148000c00 tx=0x7fa148009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:52.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.825+0000 7fa14f400700 1 -- 172.21.15.27:0/350752934 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa14801a8a0 con 0x7fa15807ca40 2024-04-16T05:31:52.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.825+0000 7fa14f400700 1 -- 172.21.15.27:0/350752934 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1056+0+0 (secure 0 0 0) 0x7fa14801ae90 con 0x7fa15807ca40 2024-04-16T05:31:52.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.825+0000 7fa14f400700 1 -- 172.21.15.27:0/350752934 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa148019e70 con 0x7fa15807ca40 2024-04-16T05:31:52.658 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.825+0000 7fa15e200700 1 -- 172.21.15.27:0/350752934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa158118320 con 0x7fa15807ca40 2024-04-16T05:31:52.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.825+0000 7fa15e200700 1 -- 172.21.15.27:0/350752934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa158118e90 con 0x7fa15807ca40 2024-04-16T05:31:52.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.826+0000 7fa14f400700 1 -- 172.21.15.27:0/350752934 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fa148019730 con 0x7fa15807ca40 2024-04-16T05:31:52.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.826+0000 7fa15e200700 1 -- 172.21.15.27:0/350752934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa15805f5f0 con 0x7fa15807ca40 2024-04-16T05:31:52.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.827+0000 7fa14f400700 1 --2- 172.21.15.27:0/350752934 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fa13c0368a0 0x7fa13c038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:52.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.827+0000 7fa14f400700 1 -- 172.21.15.27:0/350752934 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa14804a130 con 0x7fa15807ca40 2024-04-16T05:31:52.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.827+0000 7fa156a00700 1 --2- 172.21.15.27:0/350752934 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fa13c0368a0 0x7fa13c038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:52.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.828+0000 7fa156a00700 1 --2- 172.21.15.27:0/350752934 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fa13c0368a0 0x7fa13c038d50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa144009990 tx=0x7fa144006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:52.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:50.832+0000 7fa14f400700 1 -- 172.21.15.27:0/350752934 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa148021b10 con 0x7fa15807ca40 2024-04-16T05:31:52.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:51.061+0000 7fa15e200700 1 -- 172.21.15.27:0/350752934 --> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fa15807a4d0 con 0x7fa13c0368a0 2024-04-16T05:31:52.661 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:51.073+0000 7fa14f400700 1 -- 172.21.15.27:0/350752934 <== mgr.14120 v2:172.21.15.27:6800/2779081505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa15807a4d0 con 0x7fa13c0368a0 2024-04-16T05:31:52.662 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:51.075+0000 7fa15e200700 1 -- 172.21.15.27:0/350752934 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fa13c0368a0 msgr2=0x7fa13c038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:52.662 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:51.075+0000 7fa15e200700 1 --2- 172.21.15.27:0/350752934 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fa13c0368a0 0x7fa13c038d50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa144009990 tx=0x7fa144006e40 comp rx=0 tx=0).stop 2024-04-16T05:31:52.662 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:51.075+0000 7fa15e200700 1 -- 172.21.15.27:0/350752934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa15807ca40 msgr2=0x7fa158117be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:52.662 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:51.075+0000 7fa15e200700 1 --2- 172.21.15.27:0/350752934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa15807ca40 0x7fa158117be0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa148000c00 tx=0x7fa148009250 comp rx=0 tx=0).stop 2024-04-16T05:31:52.662 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:51.076+0000 7fa15e200700 1 -- 172.21.15.27:0/350752934 shutdown_connections 2024-04-16T05:31:52.662 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:51.076+0000 7fa15e200700 1 --2- 172.21.15.27:0/350752934 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fa13c0368a0 0x7fa13c038d50 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:52.662 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:51.076+0000 7fa15e200700 1 --2- 172.21.15.27:0/350752934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa15807ca40 0x7fa158117be0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:52.662 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:51.076+0000 7fa15e200700 1 -- 172.21.15.27:0/350752934 >> 172.21.15.27:0/350752934 conn(0x7fa158078350 msgr2=0x7fa158079dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:52.662 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:51.076+0000 7fa15e200700 1 -- 172.21.15.27:0/350752934 shutdown_connections 2024-04-16T05:31:52.662 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:51.076+0000 7fa15e200700 1 -- 172.21.15.27:0/350752934 wait complete. 2024-04-16T05:31:52.662 INFO:teuthology.orchestra.run.smithi027.stdout:Deploying mgr service with default placement... 2024-04-16T05:31:54.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:54 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' 2024-04-16T05:31:54.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:54 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' 2024-04-16T05:31:54.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:54 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' 2024-04-16T05:31:54.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-16T05:31:54.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.721+0000 7fe4f6400700 1 Processor -- start 2024-04-16T05:31:54.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.721+0000 7fe4f6400700 1 -- start start 2024-04-16T05:31:54.626 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.721+0000 7fe4f6400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4f0102900 0x7fe4f0102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:54.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.721+0000 7fe4f6400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f0103250 con 0x7fe4f0102900 2024-04-16T05:31:54.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.722+0000 7fe4ef400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4f0102900 0x7fe4f0102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:54.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.722+0000 7fe4ef400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4f0102900 0x7fe4f0102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54558/0 (socket says 172.21.15.27:54558) 2024-04-16T05:31:54.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.722+0000 7fe4ef400700 1 -- 172.21.15.27:0/2317026530 learned_addr learned my addr 172.21.15.27:0/2317026530 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:54.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.723+0000 7fe4ef400700 1 -- 172.21.15.27:0/2317026530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f0103390 con 0x7fe4f0102900 2024-04-16T05:31:54.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.723+0000 7fe4ef400700 1 --2- 172.21.15.27:0/2317026530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4f0102900 0x7fe4f0102d10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe4e00098e0 tx=0x7fe4e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1503432abe865c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:54.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.723+0000 7fe4ee000700 1 -- 172.21.15.27:0/2317026530 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4e000b860 con 0x7fe4f0102900 2024-04-16T05:31:54.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.723+0000 7fe4ee000700 1 -- 172.21.15.27:0/2317026530 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe4e000be90 con 0x7fe4f0102900 2024-04-16T05:31:54.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.723+0000 7fe4ee000700 1 -- 172.21.15.27:0/2317026530 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4e00103c0 con 0x7fe4f0102900 2024-04-16T05:31:54.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.724+0000 7fe4f6400700 1 -- 172.21.15.27:0/2317026530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4f0102900 msgr2=0x7fe4f0102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:54.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.724+0000 7fe4f6400700 1 --2- 172.21.15.27:0/2317026530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4f0102900 0x7fe4f0102d10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe4e00098e0 tx=0x7fe4e0009250 comp rx=0 tx=0).stop 2024-04-16T05:31:54.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.724+0000 7fe4f6400700 1 -- 172.21.15.27:0/2317026530 shutdown_connections 2024-04-16T05:31:54.627 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.724+0000 7fe4f6400700 1 --2- 172.21.15.27:0/2317026530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4f0102900 0x7fe4f0102d10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:54.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.724+0000 7fe4f6400700 1 -- 172.21.15.27:0/2317026530 >> 172.21.15.27:0/2317026530 conn(0x7fe4f00fe0a0 msgr2=0x7fe4f01004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:54.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.724+0000 7fe4f6400700 1 -- 172.21.15.27:0/2317026530 shutdown_connections 2024-04-16T05:31:54.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.725+0000 7fe4f6400700 1 -- 172.21.15.27:0/2317026530 wait complete. 2024-04-16T05:31:54.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.725+0000 7fe4f6400700 1 Processor -- start 2024-04-16T05:31:54.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.726+0000 7fe4f6400700 1 -- start start 2024-04-16T05:31:54.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.726+0000 7fe4f6400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4f0102900 0x7fe4f0113890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:54.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.726+0000 7fe4f6400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f0113dd0 con 0x7fe4f0102900 2024-04-16T05:31:54.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.726+0000 7fe4ef400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4f0102900 0x7fe4f0113890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:54.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.726+0000 7fe4ef400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4f0102900 0x7fe4f0113890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54574/0 (socket says 172.21.15.27:54574) 2024-04-16T05:31:54.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.726+0000 7fe4ef400700 1 -- 172.21.15.27:0/3866550581 learned_addr learned my addr 172.21.15.27:0/3866550581 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:54.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.727+0000 7fe4ef400700 1 -- 172.21.15.27:0/3866550581 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e00094f0 con 0x7fe4f0102900 2024-04-16T05:31:54.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.727+0000 7fe4ef400700 1 --2- 172.21.15.27:0/3866550581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4f0102900 0x7fe4f0113890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0006b00 tx=0x7fe4e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:54.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.727+0000 7fe4e7400700 1 -- 172.21.15.27:0/3866550581 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4e0010840 con 0x7fe4f0102900 2024-04-16T05:31:54.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.727+0000 7fe4e7400700 1 -- 172.21.15.27:0/3866550581 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe4e0010e70 con 0x7fe4f0102900 2024-04-16T05:31:54.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.727+0000 7fe4f6400700 1 -- 172.21.15.27:0/3866550581 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f0113fd0 con 0x7fe4f0102900 2024-04-16T05:31:54.628 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.727+0000 7fe4e7400700 1 -- 172.21.15.27:0/3866550581 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4e001ae70 con 0x7fe4f0102900 2024-04-16T05:31:54.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.728+0000 7fe4f6400700 1 -- 172.21.15.27:0/3866550581 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f0114b40 con 0x7fe4f0102900 2024-04-16T05:31:54.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.729+0000 7fe4e7400700 1 -- 172.21.15.27:0/3866550581 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fe4e001a780 con 0x7fe4f0102900 2024-04-16T05:31:54.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.729+0000 7fe4e7400700 1 --2- 172.21.15.27:0/3866550581 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fe4d40368a0 0x7fe4d4038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:54.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.730+0000 7fe4e7400700 1 -- 172.21.15.27:0/3866550581 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe4e004aac0 con 0x7fe4f0102900 2024-04-16T05:31:54.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.730+0000 7fe4f6400700 1 -- 172.21.15.27:0/3866550581 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f005f5f0 con 0x7fe4f0102900 2024-04-16T05:31:54.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.730+0000 7fe4eea00700 1 --2- 172.21.15.27:0/3866550581 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fe4d40368a0 0x7fe4d4038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:54.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.731+0000 7fe4eea00700 1 --2- 172.21.15.27:0/3866550581 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fe4d40368a0 0x7fe4d4038d50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009990 tx=0x7fe4dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:54.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.736+0000 7fe4e7400700 1 -- 172.21.15.27:0/3866550581 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fe4e001a430 con 0x7fe4f0102900 2024-04-16T05:31:54.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.970+0000 7fe4f6400700 1 -- 172.21.15.27:0/3866550581 --> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4f00ff180 con 0x7fe4d40368a0 2024-04-16T05:31:54.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.976+0000 7fe4e7400700 1 -- 172.21.15.27:0/3866550581 <== mgr.14120 v2:172.21.15.27:6800/2779081505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fe4f00ff180 con 0x7fe4d40368a0 2024-04-16T05:31:54.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.979+0000 7fe4f6400700 1 -- 172.21.15.27:0/3866550581 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fe4d40368a0 msgr2=0x7fe4d4038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:54.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.979+0000 7fe4f6400700 1 --2- 172.21.15.27:0/3866550581 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fe4d40368a0 0x7fe4d4038d50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc009990 tx=0x7fe4dc006e40 comp rx=0 tx=0).stop 2024-04-16T05:31:54.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.979+0000 7fe4f6400700 1 -- 172.21.15.27:0/3866550581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4f0102900 msgr2=0x7fe4f0113890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:54.629 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.979+0000 7fe4f6400700 1 --2- 172.21.15.27:0/3866550581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4f0102900 0x7fe4f0113890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0006b00 tx=0x7fe4e0009250 comp rx=0 tx=0).stop 2024-04-16T05:31:54.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.980+0000 7fe4f6400700 1 -- 172.21.15.27:0/3866550581 shutdown_connections 2024-04-16T05:31:54.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.980+0000 7fe4f6400700 1 --2- 172.21.15.27:0/3866550581 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fe4d40368a0 0x7fe4d4038d50 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:54.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.980+0000 7fe4f6400700 1 --2- 172.21.15.27:0/3866550581 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4f0102900 0x7fe4f0113890 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:54.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.980+0000 7fe4f6400700 1 -- 172.21.15.27:0/3866550581 >> 172.21.15.27:0/3866550581 conn(0x7fe4f00fe0a0 msgr2=0x7fe4f00fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:54.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.980+0000 7fe4f6400700 1 -- 172.21.15.27:0/3866550581 shutdown_connections 2024-04-16T05:31:54.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:53.980+0000 7fe4f6400700 1 -- 172.21.15.27:0/3866550581 wait complete. 2024-04-16T05:31:54.630 INFO:teuthology.orchestra.run.smithi027.stdout:Deploying crash service with default placement... 2024-04-16T05:31:55.471 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:55 smithi027 ceph-mon[29206]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:31:55.471 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:55 smithi027 ceph-mon[29206]: Saving service mgr spec with placement count:2 2024-04-16T05:31:56.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:56 smithi027 ceph-mon[29206]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:31:56.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:56 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' 2024-04-16T05:31:56.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-04-16T05:31:56.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.236+0000 7f57c9000700 1 Processor -- start 2024-04-16T05:31:56.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.236+0000 7f57c9000700 1 -- start start 2024-04-16T05:31:56.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.236+0000 7f57c9000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57c4102900 0x7f57c4102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:56.577 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.236+0000 7f57c9000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c4103250 con 0x7f57c4102900 2024-04-16T05:31:56.583 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.237+0000 7f57c1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57c4102900 0x7f57c4102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:56.583 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.237+0000 7f57c1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57c4102900 0x7f57c4102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55276/0 (socket says 172.21.15.27:55276) 2024-04-16T05:31:56.583 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.237+0000 7f57c1e00700 1 -- 172.21.15.27:0/2420976926 learned_addr learned my addr 172.21.15.27:0/2420976926 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:56.583 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.237+0000 7f57c1e00700 1 -- 172.21.15.27:0/2420976926 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c4103390 con 0x7f57c4102900 2024-04-16T05:31:56.583 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.238+0000 7f57c1e00700 1 --2- 172.21.15.27:0/2420976926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57c4102900 0x7f57c4102d10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f57b40098e0 tx=0x7f57b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed0f8494f758575f server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:56.583 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.238+0000 7f57c0a00700 1 -- 172.21.15.27:0/2420976926 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57b400b860 con 0x7f57c4102900 2024-04-16T05:31:56.583 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.238+0000 7f57c0a00700 1 -- 172.21.15.27:0/2420976926 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57b400be90 con 0x7f57c4102900 2024-04-16T05:31:56.583 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.238+0000 7f57c0a00700 1 -- 172.21.15.27:0/2420976926 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57b40103c0 con 0x7f57c4102900 2024-04-16T05:31:56.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.239+0000 7f57c9000700 1 -- 172.21.15.27:0/2420976926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57c4102900 msgr2=0x7f57c4102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:56.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.239+0000 7f57c9000700 1 --2- 172.21.15.27:0/2420976926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57c4102900 0x7f57c4102d10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f57b40098e0 tx=0x7f57b4009250 comp rx=0 tx=0).stop 2024-04-16T05:31:56.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.239+0000 7f57c9000700 1 -- 172.21.15.27:0/2420976926 shutdown_connections 2024-04-16T05:31:56.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.239+0000 7f57c9000700 1 --2- 172.21.15.27:0/2420976926 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57c4102900 0x7f57c4102d10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:56.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.239+0000 7f57c9000700 1 -- 172.21.15.27:0/2420976926 >> 172.21.15.27:0/2420976926 conn(0x7f57c40fe0a0 msgr2=0x7f57c41004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:56.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.239+0000 7f57c9000700 1 -- 172.21.15.27:0/2420976926 shutdown_connections 2024-04-16T05:31:56.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.239+0000 7f57c9000700 1 -- 172.21.15.27:0/2420976926 wait complete. 2024-04-16T05:31:56.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.240+0000 7f57c9000700 1 Processor -- start 2024-04-16T05:31:56.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.240+0000 7f57c9000700 1 -- start start 2024-04-16T05:31:56.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.241+0000 7f57c9000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57c4102900 0x7f57c4113850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:56.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.241+0000 7f57c9000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c4113d90 con 0x7f57c4102900 2024-04-16T05:31:56.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.241+0000 7f57c1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57c4102900 0x7f57c4113850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:56.584 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.241+0000 7f57c1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57c4102900 0x7f57c4113850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55280/0 (socket says 172.21.15.27:55280) 2024-04-16T05:31:56.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.241+0000 7f57c1e00700 1 -- 172.21.15.27:0/1088821908 learned_addr learned my addr 172.21.15.27:0/1088821908 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:56.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.241+0000 7f57c1e00700 1 -- 172.21.15.27:0/1088821908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b40094f0 con 0x7f57c4102900 2024-04-16T05:31:56.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.242+0000 7f57c1e00700 1 --2- 172.21.15.27:0/1088821908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57c4102900 0x7f57c4113850 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f57b4006b00 tx=0x7f57b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:56.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.242+0000 7f57ba000700 1 -- 172.21.15.27:0/1088821908 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57b4010840 con 0x7f57c4102900 2024-04-16T05:31:56.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.242+0000 7f57ba000700 1 -- 172.21.15.27:0/1088821908 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57b4010e70 con 0x7f57c4102900 2024-04-16T05:31:56.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.242+0000 7f57ba000700 1 -- 172.21.15.27:0/1088821908 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57b401ae70 con 0x7f57c4102900 2024-04-16T05:31:56.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.242+0000 7f57c9000700 1 -- 172.21.15.27:0/1088821908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c4113f90 con 0x7f57c4102900 2024-04-16T05:31:56.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.242+0000 7f57c9000700 1 -- 172.21.15.27:0/1088821908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c4114b00 con 0x7f57c4102900 2024-04-16T05:31:56.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.243+0000 7f57ba000700 1 -- 172.21.15.27:0/1088821908 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f57b401a780 con 0x7f57c4102900 2024-04-16T05:31:56.585 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.244+0000 7f57ba000700 1 --2- 172.21.15.27:0/1088821908 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f57a80368a0 0x7f57a8038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:56.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.244+0000 7f57c9000700 1 -- 172.21.15.27:0/1088821908 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c405f5f0 con 0x7f57c4102900 2024-04-16T05:31:56.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.244+0000 7f57ba000700 1 -- 172.21.15.27:0/1088821908 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f57b404ab80 con 0x7f57c4102900 2024-04-16T05:31:56.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.244+0000 7f57c1400700 1 --2- 172.21.15.27:0/1088821908 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f57a80368a0 0x7f57a8038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:56.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.245+0000 7f57c1400700 1 --2- 172.21.15.27:0/1088821908 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f57a80368a0 0x7f57a8038d50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009990 tx=0x7f57b0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:56.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.249+0000 7f57ba000700 1 -- 172.21.15.27:0/1088821908 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f57b401a430 con 0x7f57c4102900 2024-04-16T05:31:56.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.492+0000 7f57c9000700 1 -- 172.21.15.27:0/1088821908 --> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f57c40ff1b0 con 0x7f57a80368a0 2024-04-16T05:31:56.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.499+0000 7f57ba000700 1 -- 172.21.15.27:0/1088821908 <== mgr.14120 v2:172.21.15.27:6800/2779081505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f57c40ff1b0 con 0x7f57a80368a0 2024-04-16T05:31:56.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.501+0000 7f57c9000700 1 -- 172.21.15.27:0/1088821908 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f57a80368a0 msgr2=0x7f57a8038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:56.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.501+0000 7f57c9000700 1 --2- 172.21.15.27:0/1088821908 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f57a80368a0 0x7f57a8038d50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009990 tx=0x7f57b0006e40 comp rx=0 tx=0).stop 2024-04-16T05:31:56.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.501+0000 7f57c9000700 1 -- 172.21.15.27:0/1088821908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57c4102900 msgr2=0x7f57c4113850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:56.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.501+0000 7f57c9000700 1 --2- 172.21.15.27:0/1088821908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57c4102900 0x7f57c4113850 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f57b4006b00 tx=0x7f57b4009250 comp rx=0 tx=0).stop 2024-04-16T05:31:56.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.502+0000 7f57c9000700 1 -- 172.21.15.27:0/1088821908 shutdown_connections 2024-04-16T05:31:56.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.502+0000 7f57c9000700 1 --2- 172.21.15.27:0/1088821908 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f57a80368a0 0x7f57a8038d50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:56.586 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.502+0000 7f57c9000700 1 --2- 172.21.15.27:0/1088821908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57c4102900 0x7f57c4113850 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:56.587 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.502+0000 7f57c9000700 1 -- 172.21.15.27:0/1088821908 >> 172.21.15.27:0/1088821908 conn(0x7f57c40fe0a0 msgr2=0x7f57c40feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:56.587 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.502+0000 7f57c9000700 1 -- 172.21.15.27:0/1088821908 shutdown_connections 2024-04-16T05:31:56.587 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:55.502+0000 7f57c9000700 1 -- 172.21.15.27:0/1088821908 wait complete. 2024-04-16T05:31:56.587 INFO:teuthology.orchestra.run.smithi027.stdout:Deploying prometheus service with default placement... 2024-04-16T05:31:57.444 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:57 smithi027 ceph-mon[29206]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:31:57.444 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:57 smithi027 ceph-mon[29206]: Saving service crash spec with placement * 2024-04-16T05:31:57.444 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:57 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' 2024-04-16T05:31:58.195 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-04-16T05:31:58.195 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.198+0000 7fa6bae00700 1 Processor -- start 2024-04-16T05:31:58.195 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.198+0000 7fa6bae00700 1 -- start start 2024-04-16T05:31:58.195 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.199+0000 7fa6bae00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b41017d0 0x7fa6b4101be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:58.195 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.199+0000 7fa6bae00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b4102120 con 0x7fa6b41017d0 2024-04-16T05:31:58.196 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.199+0000 7fa6b3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b41017d0 0x7fa6b4101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:58.197 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.199+0000 7fa6b3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b41017d0 0x7fa6b4101be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55286/0 (socket says 172.21.15.27:55286) 2024-04-16T05:31:58.197 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.199+0000 7fa6b3e00700 1 -- 172.21.15.27:0/2053455516 learned_addr learned my addr 172.21.15.27:0/2053455516 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:58.197 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.200+0000 7fa6b3e00700 1 -- 172.21.15.27:0/2053455516 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b4102260 con 0x7fa6b41017d0 2024-04-16T05:31:58.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.200+0000 7fa6b3e00700 1 --2- 172.21.15.27:0/2053455516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b41017d0 0x7fa6b4101be0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400bb80 tx=0x7fa6a400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d6d180834d766ec server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:58.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.200+0000 7fa6b2a00700 1 -- 172.21.15.27:0/2053455516 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6a400dd00 con 0x7fa6b41017d0 2024-04-16T05:31:58.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.200+0000 7fa6b2a00700 1 -- 172.21.15.27:0/2053455516 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa6a400de60 con 0x7fa6b41017d0 2024-04-16T05:31:58.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.200+0000 7fa6b2a00700 1 -- 172.21.15.27:0/2053455516 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6a4018920 con 0x7fa6b41017d0 2024-04-16T05:31:58.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.201+0000 7fa6bae00700 1 -- 172.21.15.27:0/2053455516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b41017d0 msgr2=0x7fa6b4101be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:58.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.201+0000 7fa6bae00700 1 --2- 172.21.15.27:0/2053455516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b41017d0 0x7fa6b4101be0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400bb80 tx=0x7fa6a400b4f0 comp rx=0 tx=0).stop 2024-04-16T05:31:58.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.201+0000 7fa6bae00700 1 -- 172.21.15.27:0/2053455516 shutdown_connections 2024-04-16T05:31:58.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.201+0000 7fa6bae00700 1 --2- 172.21.15.27:0/2053455516 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b41017d0 0x7fa6b4101be0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:58.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.201+0000 7fa6bae00700 1 -- 172.21.15.27:0/2053455516 >> 172.21.15.27:0/2053455516 conn(0x7fa6b40fcfd0 msgr2=0x7fa6b40ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:58.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.201+0000 7fa6bae00700 1 -- 172.21.15.27:0/2053455516 shutdown_connections 2024-04-16T05:31:58.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.201+0000 7fa6bae00700 1 -- 172.21.15.27:0/2053455516 wait complete. 2024-04-16T05:31:58.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.202+0000 7fa6bae00700 1 Processor -- start 2024-04-16T05:31:58.198 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.202+0000 7fa6bae00700 1 -- start start 2024-04-16T05:31:58.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.203+0000 7fa6bae00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b41017d0 0x7fa6b410f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:58.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.203+0000 7fa6bae00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b410f8e0 con 0x7fa6b41017d0 2024-04-16T05:31:58.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.203+0000 7fa6b3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b41017d0 0x7fa6b410f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:58.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.203+0000 7fa6b3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b41017d0 0x7fa6b410f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55302/0 (socket says 172.21.15.27:55302) 2024-04-16T05:31:58.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.203+0000 7fa6b3e00700 1 -- 172.21.15.27:0/1739029556 learned_addr learned my addr 172.21.15.27:0/1739029556 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:31:58.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.203+0000 7fa6b3e00700 1 -- 172.21.15.27:0/1739029556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a400b790 con 0x7fa6b41017d0 2024-04-16T05:31:58.201 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.204+0000 7fa6b3e00700 1 --2- 172.21.15.27:0/1739029556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b41017d0 0x7fa6b410f3a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4006b00 tx=0x7fa6a400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:58.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.204+0000 7fa6abe00700 1 -- 172.21.15.27:0/1739029556 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6a4010b80 con 0x7fa6b41017d0 2024-04-16T05:31:58.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.204+0000 7fa6abe00700 1 -- 172.21.15.27:0/1739029556 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa6a401d050 con 0x7fa6b41017d0 2024-04-16T05:31:58.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.204+0000 7fa6abe00700 1 -- 172.21.15.27:0/1739029556 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6a4022590 con 0x7fa6b41017d0 2024-04-16T05:31:58.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.204+0000 7fa6bae00700 1 -- 172.21.15.27:0/1739029556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b410fae0 con 0x7fa6b41017d0 2024-04-16T05:31:58.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.204+0000 7fa6bae00700 1 -- 172.21.15.27:0/1739029556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b4110650 con 0x7fa6b41017d0 2024-04-16T05:31:58.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.205+0000 7fa6abe00700 1 -- 172.21.15.27:0/1739029556 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fa6a4010ce0 con 0x7fa6b41017d0 2024-04-16T05:31:58.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.205+0000 7fa6bae00700 1 -- 172.21.15.27:0/1739029556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b405f5f0 con 0x7fa6b41017d0 2024-04-16T05:31:58.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.206+0000 7fa6abe00700 1 --2- 172.21.15.27:0/1739029556 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fa6980368a0 0x7fa698038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:31:58.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.206+0000 7fa6abe00700 1 -- 172.21.15.27:0/1739029556 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa6a4014070 con 0x7fa6b41017d0 2024-04-16T05:31:58.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.208+0000 7fa6b3400700 1 --2- 172.21.15.27:0/1739029556 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fa6980368a0 0x7fa698038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:31:58.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.209+0000 7fa6b3400700 1 --2- 172.21.15.27:0/1739029556 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fa6980368a0 0x7fa698038d50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0009990 tx=0x7fa6a0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:31:58.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.211+0000 7fa6abe00700 1 -- 172.21.15.27:0/1739029556 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa6a40226f0 con 0x7fa6b41017d0 2024-04-16T05:31:58.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.445+0000 7fa6bae00700 1 -- 172.21.15.27:0/1739029556 --> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6b4110a00 con 0x7fa6980368a0 2024-04-16T05:31:58.202 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.452+0000 7fa6abe00700 1 -- 172.21.15.27:0/1739029556 <== mgr.14120 v2:172.21.15.27:6800/2779081505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fa6b4110a00 con 0x7fa6980368a0 2024-04-16T05:31:58.203 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.455+0000 7fa6bae00700 1 -- 172.21.15.27:0/1739029556 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fa6980368a0 msgr2=0x7fa698038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:58.203 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.455+0000 7fa6bae00700 1 --2- 172.21.15.27:0/1739029556 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fa6980368a0 0x7fa698038d50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0009990 tx=0x7fa6a0006e40 comp rx=0 tx=0).stop 2024-04-16T05:31:58.204 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.455+0000 7fa6bae00700 1 -- 172.21.15.27:0/1739029556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b41017d0 msgr2=0x7fa6b410f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:31:58.204 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.455+0000 7fa6bae00700 1 --2- 172.21.15.27:0/1739029556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b41017d0 0x7fa6b410f3a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4006b00 tx=0x7fa6a400b4f0 comp rx=0 tx=0).stop 2024-04-16T05:31:58.204 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.455+0000 7fa6bae00700 1 -- 172.21.15.27:0/1739029556 shutdown_connections 2024-04-16T05:31:58.204 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.455+0000 7fa6bae00700 1 --2- 172.21.15.27:0/1739029556 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fa6980368a0 0x7fa698038d50 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:58.204 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.455+0000 7fa6bae00700 1 --2- 172.21.15.27:0/1739029556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa6b41017d0 0x7fa6b410f3a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:31:58.204 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.455+0000 7fa6bae00700 1 -- 172.21.15.27:0/1739029556 >> 172.21.15.27:0/1739029556 conn(0x7fa6b40fcfd0 msgr2=0x7fa6b4105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:31:58.204 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.456+0000 7fa6bae00700 1 -- 172.21.15.27:0/1739029556 shutdown_connections 2024-04-16T05:31:58.204 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:57.456+0000 7fa6bae00700 1 -- 172.21.15.27:0/1739029556 wait complete. 2024-04-16T05:31:58.204 INFO:teuthology.orchestra.run.smithi027.stdout:Deploying grafana service with default placement... 2024-04-16T05:31:58.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:58 smithi027 ceph-mon[29206]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:31:58.511 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:58 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' 2024-04-16T05:31:59.320 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:59 smithi027 ceph-mon[29206]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:31:59.320 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:59 smithi027 ceph-mon[29206]: Saving service prometheus spec with placement count:1 2024-04-16T05:31:59.320 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:59 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' 2024-04-16T05:31:59.320 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:31:59 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:00.293 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:00 smithi027 ceph-mon[29206]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:32:00.293 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:00 smithi027 ceph-mon[29206]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:32:00.293 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:00 smithi027 ceph-mon[29206]: Saving service grafana spec with placement count:1 2024-04-16T05:32:00.799 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-04-16T05:32:00.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.802+0000 7f0be4000700 1 Processor -- start 2024-04-16T05:32:00.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.802+0000 7f0be4000700 1 -- start start 2024-04-16T05:32:00.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.802+0000 7f0be4000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bdc104c30 0x7f0bdc105040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:00.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.802+0000 7f0be4000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc105610 con 0x7f0bdc104c30 2024-04-16T05:32:00.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.803+0000 7f0be1600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bdc104c30 0x7f0bdc105040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:00.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.803+0000 7f0be1600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bdc104c30 0x7f0bdc105040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55304/0 (socket says 172.21.15.27:55304) 2024-04-16T05:32:00.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.803+0000 7f0be1600700 1 -- 172.21.15.27:0/248268867 learned_addr learned my addr 172.21.15.27:0/248268867 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:00.800 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.803+0000 7f0be1600700 1 -- 172.21.15.27:0/248268867 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bdc105750 con 0x7f0bdc104c30 2024-04-16T05:32:00.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.804+0000 7f0be1600700 1 --2- 172.21.15.27:0/248268867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bdc104c30 0x7f0bdc105040 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0bd00098e0 tx=0x7f0bd0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3852ea4d35de1ffa server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:00.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.804+0000 7f0bdbe00700 1 -- 172.21.15.27:0/248268867 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bd000b860 con 0x7f0bdc104c30 2024-04-16T05:32:00.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.804+0000 7f0bdbe00700 1 -- 172.21.15.27:0/248268867 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0bd000be90 con 0x7f0bdc104c30 2024-04-16T05:32:00.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.804+0000 7f0bdbe00700 1 -- 172.21.15.27:0/248268867 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bd00103c0 con 0x7f0bdc104c30 2024-04-16T05:32:00.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.805+0000 7f0be4000700 1 -- 172.21.15.27:0/248268867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bdc104c30 msgr2=0x7f0bdc105040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:00.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.805+0000 7f0be4000700 1 --2- 172.21.15.27:0/248268867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bdc104c30 0x7f0bdc105040 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0bd00098e0 tx=0x7f0bd0009250 comp rx=0 tx=0).stop 2024-04-16T05:32:00.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.805+0000 7f0be4000700 1 -- 172.21.15.27:0/248268867 shutdown_connections 2024-04-16T05:32:00.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.805+0000 7f0be4000700 1 --2- 172.21.15.27:0/248268867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bdc104c30 0x7f0bdc105040 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:00.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.805+0000 7f0be4000700 1 -- 172.21.15.27:0/248268867 >> 172.21.15.27:0/248268867 conn(0x7f0bdc100590 msgr2=0x7f0bdc102970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:00.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.805+0000 7f0be4000700 1 -- 172.21.15.27:0/248268867 shutdown_connections 2024-04-16T05:32:00.801 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.806+0000 7f0be4000700 1 -- 172.21.15.27:0/248268867 wait complete. 2024-04-16T05:32:00.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.807+0000 7f0be4000700 1 Processor -- start 2024-04-16T05:32:00.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.807+0000 7f0be4000700 1 -- start start 2024-04-16T05:32:00.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.807+0000 7f0be4000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bdc104c30 0x7f0bdc1137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:00.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.807+0000 7f0be4000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc113cf0 con 0x7f0bdc104c30 2024-04-16T05:32:00.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.807+0000 7f0be1600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bdc104c30 0x7f0bdc1137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:00.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.807+0000 7f0be1600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bdc104c30 0x7f0bdc1137b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55310/0 (socket says 172.21.15.27:55310) 2024-04-16T05:32:00.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.807+0000 7f0be1600700 1 -- 172.21.15.27:0/3044405255 learned_addr learned my addr 172.21.15.27:0/3044405255 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:00.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.808+0000 7f0be1600700 1 -- 172.21.15.27:0/3044405255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd00094f0 con 0x7f0bdc104c30 2024-04-16T05:32:00.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.808+0000 7f0be1600700 1 --2- 172.21.15.27:0/3044405255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bdc104c30 0x7f0bdc1137b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0006b00 tx=0x7f0bd0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:00.802 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.808+0000 7f0bd9600700 1 -- 172.21.15.27:0/3044405255 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bd0010840 con 0x7f0bdc104c30 2024-04-16T05:32:00.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.808+0000 7f0bd9600700 1 -- 172.21.15.27:0/3044405255 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0bd0010e70 con 0x7f0bdc104c30 2024-04-16T05:32:00.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.809+0000 7f0be4000700 1 -- 172.21.15.27:0/3044405255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bdc113ef0 con 0x7f0bdc104c30 2024-04-16T05:32:00.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.809+0000 7f0bd9600700 1 -- 172.21.15.27:0/3044405255 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bd001ae70 con 0x7f0bdc104c30 2024-04-16T05:32:00.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.809+0000 7f0be4000700 1 -- 172.21.15.27:0/3044405255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bdc114a60 con 0x7f0bdc104c30 2024-04-16T05:32:00.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.810+0000 7f0bd9600700 1 -- 172.21.15.27:0/3044405255 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f0bd00244d0 con 0x7f0bdc104c30 2024-04-16T05:32:00.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.810+0000 7f0bd9600700 1 --2- 172.21.15.27:0/3044405255 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f0bc4036850 0x7f0bc4038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:00.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.810+0000 7f0bd9600700 1 -- 172.21.15.27:0/3044405255 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f0bd004ac50 con 0x7f0bdc104c30 2024-04-16T05:32:00.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.810+0000 7f0be4000700 1 -- 172.21.15.27:0/3044405255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bdc05f5f0 con 0x7f0bdc104c30 2024-04-16T05:32:00.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.811+0000 7f0be0c00700 1 --2- 172.21.15.27:0/3044405255 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f0bc4036850 0x7f0bc4038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:00.803 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.811+0000 7f0be0c00700 1 --2- 172.21.15.27:0/3044405255 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f0bc4036850 0x7f0bc4038d00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8009990 tx=0x7f0bc8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:00.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:58.818+0000 7f0bd9600700 1 -- 172.21.15.27:0/3044405255 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f0bd002e440 con 0x7f0bdc104c30 2024-04-16T05:32:00.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:59.060+0000 7f0be4000700 1 -- 172.21.15.27:0/3044405255 --> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bdc102690 con 0x7f0bc4036850 2024-04-16T05:32:00.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:59.071+0000 7f0bd9600700 1 -- 172.21.15.27:0/3044405255 <== mgr.14120 v2:172.21.15.27:6800/2779081505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f0bdc102690 con 0x7f0bc4036850 2024-04-16T05:32:00.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:59.073+0000 7f0be4000700 1 -- 172.21.15.27:0/3044405255 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f0bc4036850 msgr2=0x7f0bc4038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:00.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:59.073+0000 7f0be4000700 1 --2- 172.21.15.27:0/3044405255 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f0bc4036850 0x7f0bc4038d00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8009990 tx=0x7f0bc8006e40 comp rx=0 tx=0).stop 2024-04-16T05:32:00.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:59.073+0000 7f0be4000700 1 -- 172.21.15.27:0/3044405255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bdc104c30 msgr2=0x7f0bdc1137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:00.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:59.073+0000 7f0be4000700 1 --2- 172.21.15.27:0/3044405255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bdc104c30 0x7f0bdc1137b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0006b00 tx=0x7f0bd0009250 comp rx=0 tx=0).stop 2024-04-16T05:32:00.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:59.074+0000 7f0be4000700 1 -- 172.21.15.27:0/3044405255 shutdown_connections 2024-04-16T05:32:00.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:59.074+0000 7f0be4000700 1 --2- 172.21.15.27:0/3044405255 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f0bc4036850 0x7f0bc4038d00 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:00.804 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:59.074+0000 7f0be4000700 1 --2- 172.21.15.27:0/3044405255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0bdc104c30 0x7f0bdc1137b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:00.805 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:59.074+0000 7f0be4000700 1 -- 172.21.15.27:0/3044405255 >> 172.21.15.27:0/3044405255 conn(0x7f0bdc100590 msgr2=0x7f0bdc101f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:00.805 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:59.074+0000 7f0be4000700 1 -- 172.21.15.27:0/3044405255 shutdown_connections 2024-04-16T05:32:00.805 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:31:59.074+0000 7f0be4000700 1 -- 172.21.15.27:0/3044405255 wait complete. 2024-04-16T05:32:00.805 INFO:teuthology.orchestra.run.smithi027.stdout:Deploying node-exporter service with default placement... 2024-04-16T05:32:02.359 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:02 smithi027 ceph-mon[29206]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:32:02.360 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:02 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:02.374 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-04-16T05:32:02.374 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.428+0000 7fccf8c00700 1 Processor -- start 2024-04-16T05:32:02.374 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.428+0000 7fccf8c00700 1 -- start start 2024-04-16T05:32:02.374 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.429+0000 7fccf8c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fccf4104bf0 0x7fccf4105000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:02.374 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.429+0000 7fccf8c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf41055d0 con 0x7fccf4104bf0 2024-04-16T05:32:02.374 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.429+0000 7fccf1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fccf4104bf0 0x7fccf4105000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:02.374 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.429+0000 7fccf1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fccf4104bf0 0x7fccf4105000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55312/0 (socket says 172.21.15.27:55312) 2024-04-16T05:32:02.375 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.429+0000 7fccf1e00700 1 -- 172.21.15.27:0/436113070 learned_addr learned my addr 172.21.15.27:0/436113070 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:02.375 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.430+0000 7fccf1e00700 1 -- 172.21.15.27:0/436113070 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf4105710 con 0x7fccf4104bf0 2024-04-16T05:32:02.375 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.430+0000 7fccf1e00700 1 --2- 172.21.15.27:0/436113070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fccf4104bf0 0x7fccf4105000 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcce40098e0 tx=0x7fcce4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d57189d6cc6055d server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:02.375 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.430+0000 7fccf0a00700 1 -- 172.21.15.27:0/436113070 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcce400b860 con 0x7fccf4104bf0 2024-04-16T05:32:02.375 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.430+0000 7fccf0a00700 1 -- 172.21.15.27:0/436113070 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcce400be90 con 0x7fccf4104bf0 2024-04-16T05:32:02.375 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.430+0000 7fccf0a00700 1 -- 172.21.15.27:0/436113070 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcce40103c0 con 0x7fccf4104bf0 2024-04-16T05:32:02.375 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.431+0000 7fccf8c00700 1 -- 172.21.15.27:0/436113070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fccf4104bf0 msgr2=0x7fccf4105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:02.375 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.431+0000 7fccf8c00700 1 --2- 172.21.15.27:0/436113070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fccf4104bf0 0x7fccf4105000 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcce40098e0 tx=0x7fcce4009250 comp rx=0 tx=0).stop 2024-04-16T05:32:02.375 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.431+0000 7fccf8c00700 1 -- 172.21.15.27:0/436113070 shutdown_connections 2024-04-16T05:32:02.375 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.431+0000 7fccf8c00700 1 --2- 172.21.15.27:0/436113070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fccf4104bf0 0x7fccf4105000 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:02.375 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.431+0000 7fccf8c00700 1 -- 172.21.15.27:0/436113070 >> 172.21.15.27:0/436113070 conn(0x7fccf4078320 msgr2=0x7fccf4078720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:02.375 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.431+0000 7fccf8c00700 1 -- 172.21.15.27:0/436113070 shutdown_connections 2024-04-16T05:32:02.375 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.431+0000 7fccf8c00700 1 -- 172.21.15.27:0/436113070 wait complete. 2024-04-16T05:32:02.375 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.432+0000 7fccf8c00700 1 Processor -- start 2024-04-16T05:32:02.375 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.432+0000 7fccf8c00700 1 -- start start 2024-04-16T05:32:02.376 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.432+0000 7fccf8c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fccf4104bf0 0x7fccf41137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:02.376 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.432+0000 7fccf8c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf4113cf0 con 0x7fccf4104bf0 2024-04-16T05:32:02.376 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.433+0000 7fccf1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fccf4104bf0 0x7fccf41137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:02.376 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.433+0000 7fccf1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fccf4104bf0 0x7fccf41137b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55316/0 (socket says 172.21.15.27:55316) 2024-04-16T05:32:02.376 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.433+0000 7fccf1e00700 1 -- 172.21.15.27:0/1082329863 learned_addr learned my addr 172.21.15.27:0/1082329863 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:02.376 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.433+0000 7fccf1e00700 1 -- 172.21.15.27:0/1082329863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce40094f0 con 0x7fccf4104bf0 2024-04-16T05:32:02.376 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.433+0000 7fccf1e00700 1 --2- 172.21.15.27:0/1082329863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fccf4104bf0 0x7fccf41137b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcce40090b0 tx=0x7fcce4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:02.376 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.434+0000 7fccea000700 1 -- 172.21.15.27:0/1082329863 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcce4010840 con 0x7fccf4104bf0 2024-04-16T05:32:02.376 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.434+0000 7fccea000700 1 -- 172.21.15.27:0/1082329863 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcce4010e70 con 0x7fccf4104bf0 2024-04-16T05:32:02.376 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.434+0000 7fccea000700 1 -- 172.21.15.27:0/1082329863 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcce4024e40 con 0x7fccf4104bf0 2024-04-16T05:32:02.376 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.434+0000 7fccf8c00700 1 -- 172.21.15.27:0/1082329863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf4113ef0 con 0x7fccf4104bf0 2024-04-16T05:32:02.376 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.434+0000 7fccf8c00700 1 -- 172.21.15.27:0/1082329863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf4114a60 con 0x7fccf4104bf0 2024-04-16T05:32:02.376 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.435+0000 7fccea000700 1 -- 172.21.15.27:0/1082329863 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fcce4024430 con 0x7fccf4104bf0 2024-04-16T05:32:02.376 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.435+0000 7fccf8c00700 1 -- 172.21.15.27:0/1082329863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf405f5f0 con 0x7fccf4104bf0 2024-04-16T05:32:02.376 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.435+0000 7fccea000700 1 --2- 172.21.15.27:0/1082329863 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fccd8036850 0x7fccd8038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:02.376 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.435+0000 7fccea000700 1 -- 172.21.15.27:0/1082329863 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fcce404a740 con 0x7fccf4104bf0 2024-04-16T05:32:02.377 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.436+0000 7fccf1400700 1 --2- 172.21.15.27:0/1082329863 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fccd8036850 0x7fccd8038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:02.377 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.436+0000 7fccf1400700 1 --2- 172.21.15.27:0/1082329863 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fccd8036850 0x7fccd8038d00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fcce0009990 tx=0x7fcce0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:02.377 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.440+0000 7fccea000700 1 -- 172.21.15.27:0/1082329863 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fcce402c930 con 0x7fccf4104bf0 2024-04-16T05:32:02.377 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.668+0000 7fccf8c00700 1 -- 172.21.15.27:0/1082329863 --> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fccf4102b70 con 0x7fccd8036850 2024-04-16T05:32:02.377 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.677+0000 7fccea000700 1 -- 172.21.15.27:0/1082329863 <== mgr.14120 v2:172.21.15.27:6800/2779081505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fccf4102b70 con 0x7fccd8036850 2024-04-16T05:32:02.377 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.680+0000 7fccf8c00700 1 -- 172.21.15.27:0/1082329863 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fccd8036850 msgr2=0x7fccd8038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:02.377 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.680+0000 7fccf8c00700 1 --2- 172.21.15.27:0/1082329863 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fccd8036850 0x7fccd8038d00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fcce0009990 tx=0x7fcce0006e40 comp rx=0 tx=0).stop 2024-04-16T05:32:02.377 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.680+0000 7fccf8c00700 1 -- 172.21.15.27:0/1082329863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fccf4104bf0 msgr2=0x7fccf41137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:02.377 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.680+0000 7fccf8c00700 1 --2- 172.21.15.27:0/1082329863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fccf4104bf0 0x7fccf41137b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcce40090b0 tx=0x7fcce4009250 comp rx=0 tx=0).stop 2024-04-16T05:32:02.377 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.681+0000 7fccf8c00700 1 -- 172.21.15.27:0/1082329863 shutdown_connections 2024-04-16T05:32:02.377 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.681+0000 7fccf8c00700 1 --2- 172.21.15.27:0/1082329863 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fccd8036850 0x7fccd8038d00 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:02.377 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.681+0000 7fccf8c00700 1 --2- 172.21.15.27:0/1082329863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fccf4104bf0 0x7fccf41137b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:02.377 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.681+0000 7fccf8c00700 1 -- 172.21.15.27:0/1082329863 >> 172.21.15.27:0/1082329863 conn(0x7fccf4078320 msgr2=0x7fccf4102460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:02.377 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.681+0000 7fccf8c00700 1 -- 172.21.15.27:0/1082329863 shutdown_connections 2024-04-16T05:32:02.377 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:01.681+0000 7fccf8c00700 1 -- 172.21.15.27:0/1082329863 wait complete. 2024-04-16T05:32:02.378 INFO:teuthology.orchestra.run.smithi027.stdout:Deploying alertmanager service with default placement... 2024-04-16T05:32:03.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:03 smithi027 ceph-mon[29206]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:32:03.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:03 smithi027 ceph-mon[29206]: Saving service node-exporter spec with placement * 2024-04-16T05:32:04.025 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-04-16T05:32:04.026 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.041+0000 7fb08d600700 1 Processor -- start 2024-04-16T05:32:04.026 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.041+0000 7fb08d600700 1 -- start start 2024-04-16T05:32:04.026 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.042+0000 7fb08d600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08807ca40 0x7fb08807ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:04.026 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.042+0000 7fb08d600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08807d420 con 0x7fb08807ca40 2024-04-16T05:32:04.027 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.042+0000 7fb086a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08807ca40 0x7fb08807ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:04.027 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.042+0000 7fb086a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08807ca40 0x7fb08807ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55318/0 (socket says 172.21.15.27:55318) 2024-04-16T05:32:04.027 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.042+0000 7fb086a00700 1 -- 172.21.15.27:0/520381559 learned_addr learned my addr 172.21.15.27:0/520381559 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:04.027 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.043+0000 7fb086a00700 1 -- 172.21.15.27:0/520381559 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb08807d560 con 0x7fb08807ca40 2024-04-16T05:32:04.027 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.043+0000 7fb086a00700 1 --2- 172.21.15.27:0/520381559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08807ca40 0x7fb08807ce50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb0780098e0 tx=0x7fb078009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e58e6b363c9720 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:04.028 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.043+0000 7fb085600700 1 -- 172.21.15.27:0/520381559 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb07800b860 con 0x7fb08807ca40 2024-04-16T05:32:04.028 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.043+0000 7fb085600700 1 -- 172.21.15.27:0/520381559 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb07800be90 con 0x7fb08807ca40 2024-04-16T05:32:04.028 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.043+0000 7fb085600700 1 -- 172.21.15.27:0/520381559 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0780103c0 con 0x7fb08807ca40 2024-04-16T05:32:04.028 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.044+0000 7fb08d600700 1 -- 172.21.15.27:0/520381559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08807ca40 msgr2=0x7fb08807ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:04.029 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.044+0000 7fb08d600700 1 --2- 172.21.15.27:0/520381559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08807ca40 0x7fb08807ce50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb0780098e0 tx=0x7fb078009250 comp rx=0 tx=0).stop 2024-04-16T05:32:04.029 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.044+0000 7fb08d600700 1 -- 172.21.15.27:0/520381559 shutdown_connections 2024-04-16T05:32:04.029 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.044+0000 7fb08d600700 1 --2- 172.21.15.27:0/520381559 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08807ca40 0x7fb08807ce50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:04.030 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.044+0000 7fb08d600700 1 -- 172.21.15.27:0/520381559 >> 172.21.15.27:0/520381559 conn(0x7fb088078350 msgr2=0x7fb08807a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:04.030 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.044+0000 7fb08d600700 1 -- 172.21.15.27:0/520381559 shutdown_connections 2024-04-16T05:32:04.031 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.044+0000 7fb08d600700 1 -- 172.21.15.27:0/520381559 wait complete. 2024-04-16T05:32:04.031 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.045+0000 7fb08d600700 1 Processor -- start 2024-04-16T05:32:04.031 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.045+0000 7fb08d600700 1 -- start start 2024-04-16T05:32:04.031 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.046+0000 7fb08d600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08807ca40 0x7fb088117c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:04.031 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.046+0000 7fb08d600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb088118150 con 0x7fb08807ca40 2024-04-16T05:32:04.031 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.046+0000 7fb086a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08807ca40 0x7fb088117c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:04.031 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.046+0000 7fb086a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08807ca40 0x7fb088117c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55334/0 (socket says 172.21.15.27:55334) 2024-04-16T05:32:04.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.046+0000 7fb086a00700 1 -- 172.21.15.27:0/3852075881 learned_addr learned my addr 172.21.15.27:0/3852075881 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:04.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.046+0000 7fb086a00700 1 -- 172.21.15.27:0/3852075881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0780094f0 con 0x7fb08807ca40 2024-04-16T05:32:04.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.047+0000 7fb086a00700 1 --2- 172.21.15.27:0/3852075881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08807ca40 0x7fb088117c10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb078006b00 tx=0x7fb078009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:04.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.047+0000 7fb07ea00700 1 -- 172.21.15.27:0/3852075881 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb078010840 con 0x7fb08807ca40 2024-04-16T05:32:04.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.047+0000 7fb07ea00700 1 -- 172.21.15.27:0/3852075881 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb078010e70 con 0x7fb08807ca40 2024-04-16T05:32:04.032 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.047+0000 7fb08d600700 1 -- 172.21.15.27:0/3852075881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb088118350 con 0x7fb08807ca40 2024-04-16T05:32:04.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.047+0000 7fb07ea00700 1 -- 172.21.15.27:0/3852075881 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb07801ae70 con 0x7fb08807ca40 2024-04-16T05:32:04.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.047+0000 7fb08d600700 1 -- 172.21.15.27:0/3852075881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb088118ec0 con 0x7fb08807ca40 2024-04-16T05:32:04.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.048+0000 7fb07ea00700 1 -- 172.21.15.27:0/3852075881 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7fb07801a780 con 0x7fb08807ca40 2024-04-16T05:32:04.033 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.048+0000 7fb08d600700 1 -- 172.21.15.27:0/3852075881 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb08805f5f0 con 0x7fb08807ca40 2024-04-16T05:32:04.034 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.049+0000 7fb07ea00700 1 --2- 172.21.15.27:0/3852075881 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fb06c0368a0 0x7fb06c038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:04.034 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.049+0000 7fb07ea00700 1 -- 172.21.15.27:0/3852075881 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb07804aba0 con 0x7fb08807ca40 2024-04-16T05:32:04.035 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.049+0000 7fb086000700 1 --2- 172.21.15.27:0/3852075881 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fb06c0368a0 0x7fb06c038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:04.035 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.050+0000 7fb086000700 1 --2- 172.21.15.27:0/3852075881 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fb06c0368a0 0x7fb06c038d50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb074009990 tx=0x7fb074006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:04.035 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.054+0000 7fb07ea00700 1 -- 172.21.15.27:0/3852075881 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb07801a430 con 0x7fb08807ca40 2024-04-16T05:32:04.035 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.279+0000 7fb08d600700 1 -- 172.21.15.27:0/3852075881 --> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fb08807a4d0 con 0x7fb06c0368a0 2024-04-16T05:32:04.035 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.288+0000 7fb07ea00700 1 -- 172.21.15.27:0/3852075881 <== mgr.14120 v2:172.21.15.27:6800/2779081505 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fb08807a4d0 con 0x7fb06c0368a0 2024-04-16T05:32:04.035 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.291+0000 7fb08d600700 1 -- 172.21.15.27:0/3852075881 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fb06c0368a0 msgr2=0x7fb06c038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:04.035 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.291+0000 7fb08d600700 1 --2- 172.21.15.27:0/3852075881 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fb06c0368a0 0x7fb06c038d50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb074009990 tx=0x7fb074006e40 comp rx=0 tx=0).stop 2024-04-16T05:32:04.036 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.291+0000 7fb08d600700 1 -- 172.21.15.27:0/3852075881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08807ca40 msgr2=0x7fb088117c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:04.036 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.291+0000 7fb08d600700 1 --2- 172.21.15.27:0/3852075881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08807ca40 0x7fb088117c10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb078006b00 tx=0x7fb078009250 comp rx=0 tx=0).stop 2024-04-16T05:32:04.036 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.291+0000 7fb08d600700 1 -- 172.21.15.27:0/3852075881 shutdown_connections 2024-04-16T05:32:04.036 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.291+0000 7fb08d600700 1 --2- 172.21.15.27:0/3852075881 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7fb06c0368a0 0x7fb06c038d50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:04.036 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.291+0000 7fb08d600700 1 --2- 172.21.15.27:0/3852075881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb08807ca40 0x7fb088117c10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:04.037 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.291+0000 7fb08d600700 1 -- 172.21.15.27:0/3852075881 >> 172.21.15.27:0/3852075881 conn(0x7fb088078350 msgr2=0x7fb088079dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:04.038 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.291+0000 7fb08d600700 1 -- 172.21.15.27:0/3852075881 shutdown_connections 2024-04-16T05:32:04.039 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:03.291+0000 7fb08d600700 1 -- 172.21.15.27:0/3852075881 wait complete. 2024-04-16T05:32:04.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:04 smithi027 ceph-mon[29206]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:32:04.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:04 smithi027 ceph-mon[29206]: from='mgr.14120 172.21.15.27:0/3455166156' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:05.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:05 smithi027 ceph-mon[29206]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:32:05.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:05 smithi027 ceph-mon[29206]: Saving service alertmanager spec with placement count:1 2024-04-16T05:32:05.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:05 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/406332860' entity='client.admin' 2024-04-16T05:32:05.617 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.672+0000 7f57e7000700 1 Processor -- start 2024-04-16T05:32:05.617 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.672+0000 7f57e7000700 1 -- start start 2024-04-16T05:32:05.617 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.673+0000 7f57e7000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e0102900 0x7f57e0102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:05.617 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.673+0000 7f57e7000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e0103250 con 0x7f57e0102900 2024-04-16T05:32:05.617 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.674+0000 7f57dfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e0102900 0x7f57e0102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:05.617 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.674+0000 7f57dfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e0102900 0x7f57e0102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40742/0 (socket says 172.21.15.27:40742) 2024-04-16T05:32:05.617 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.674+0000 7f57dfe00700 1 -- 172.21.15.27:0/2506998707 learned_addr learned my addr 172.21.15.27:0/2506998707 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:05.622 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.674+0000 7f57dfe00700 1 -- 172.21.15.27:0/2506998707 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e0103390 con 0x7f57e0102900 2024-04-16T05:32:05.622 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.675+0000 7f57dfe00700 1 --2- 172.21.15.27:0/2506998707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e0102900 0x7f57e0102d10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f57d0009e50 tx=0x7f57d0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34437653a10da78b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:05.622 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.675+0000 7f57df400700 1 -- 172.21.15.27:0/2506998707 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57d00092e0 con 0x7f57e0102900 2024-04-16T05:32:05.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.675+0000 7f57df400700 1 -- 172.21.15.27:0/2506998707 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57d0012430 con 0x7f57e0102900 2024-04-16T05:32:05.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.675+0000 7f57df400700 1 -- 172.21.15.27:0/2506998707 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57d00075a0 con 0x7f57e0102900 2024-04-16T05:32:05.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.675+0000 7f57e7000700 1 -- 172.21.15.27:0/2506998707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e0102900 msgr2=0x7f57e0102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:05.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.675+0000 7f57e7000700 1 --2- 172.21.15.27:0/2506998707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e0102900 0x7f57e0102d10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f57d0009e50 tx=0x7f57d0009840 comp rx=0 tx=0).stop 2024-04-16T05:32:05.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.676+0000 7f57e7000700 1 -- 172.21.15.27:0/2506998707 shutdown_connections 2024-04-16T05:32:05.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.676+0000 7f57e7000700 1 --2- 172.21.15.27:0/2506998707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e0102900 0x7f57e0102d10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:05.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.676+0000 7f57e7000700 1 -- 172.21.15.27:0/2506998707 >> 172.21.15.27:0/2506998707 conn(0x7f57e00fe0a0 msgr2=0x7f57e01004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:05.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.676+0000 7f57e7000700 1 -- 172.21.15.27:0/2506998707 shutdown_connections 2024-04-16T05:32:05.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.676+0000 7f57e7000700 1 -- 172.21.15.27:0/2506998707 wait complete. 2024-04-16T05:32:05.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.677+0000 7f57e7000700 1 Processor -- start 2024-04-16T05:32:05.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.677+0000 7f57e7000700 1 -- start start 2024-04-16T05:32:05.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.677+0000 7f57e7000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e0102900 0x7f57e010f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:05.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.677+0000 7f57e7000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e010fa00 con 0x7f57e0102900 2024-04-16T05:32:05.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.678+0000 7f57dfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e0102900 0x7f57e010f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:05.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.678+0000 7f57dfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e0102900 0x7f57e010f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40756/0 (socket says 172.21.15.27:40756) 2024-04-16T05:32:05.623 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.678+0000 7f57dfe00700 1 -- 172.21.15.27:0/406332860 learned_addr learned my addr 172.21.15.27:0/406332860 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:05.624 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.678+0000 7f57dfe00700 1 -- 172.21.15.27:0/406332860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d0009ae0 con 0x7f57e0102900 2024-04-16T05:32:05.624 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.679+0000 7f57dfe00700 1 --2- 172.21.15.27:0/406332860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e0102900 0x7f57e010f4c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f57d0012de0 tx=0x7f57d0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:05.624 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.679+0000 7f57dcc00700 1 -- 172.21.15.27:0/406332860 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57d00092e0 con 0x7f57e0102900 2024-04-16T05:32:05.624 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.679+0000 7f57e7000700 1 -- 172.21.15.27:0/406332860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e010fc00 con 0x7f57e0102900 2024-04-16T05:32:05.624 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.679+0000 7f57dcc00700 1 -- 172.21.15.27:0/406332860 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57d0007ce0 con 0x7f57e0102900 2024-04-16T05:32:05.624 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.679+0000 7f57dcc00700 1 -- 172.21.15.27:0/406332860 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57d0025930 con 0x7f57e0102900 2024-04-16T05:32:05.624 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.679+0000 7f57e7000700 1 -- 172.21.15.27:0/406332860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e0110770 con 0x7f57e0102900 2024-04-16T05:32:05.624 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.680+0000 7f57dcc00700 1 -- 172.21.15.27:0/406332860 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f57d0019050 con 0x7f57e0102900 2024-04-16T05:32:05.624 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.681+0000 7f57e7000700 1 -- 172.21.15.27:0/406332860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57e005f5f0 con 0x7f57e0102900 2024-04-16T05:32:05.624 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.681+0000 7f57dcc00700 1 --2- 172.21.15.27:0/406332860 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f57c403acd0 0x7f57c403d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:05.624 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.681+0000 7f57dcc00700 1 -- 172.21.15.27:0/406332860 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f57d004b8b0 con 0x7f57e0102900 2024-04-16T05:32:05.624 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.681+0000 7f57d7400700 1 --2- 172.21.15.27:0/406332860 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f57c403acd0 0x7f57c403d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:05.624 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.682+0000 7f57d7400700 1 --2- 172.21.15.27:0/406332860 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f57c403acd0 0x7f57c403d180 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f57c8009990 tx=0x7f57c8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:05.624 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.686+0000 7f57dcc00700 1 -- 172.21.15.27:0/406332860 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f57d0007810 con 0x7f57e0102900 2024-04-16T05:32:05.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.904+0000 7f57e7000700 1 -- 172.21.15.27:0/406332860 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f57e004bed0 con 0x7f57e0102900 2024-04-16T05:32:05.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.911+0000 7f57dcc00700 1 -- 172.21.15.27:0/406332860 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f57d0020020 con 0x7f57e0102900 2024-04-16T05:32:05.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.914+0000 7f57e7000700 1 -- 172.21.15.27:0/406332860 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f57c403acd0 msgr2=0x7f57c403d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:05.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.914+0000 7f57e7000700 1 --2- 172.21.15.27:0/406332860 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f57c403acd0 0x7f57c403d180 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f57c8009990 tx=0x7f57c8006e40 comp rx=0 tx=0).stop 2024-04-16T05:32:05.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.914+0000 7f57e7000700 1 -- 172.21.15.27:0/406332860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e0102900 msgr2=0x7f57e010f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:05.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.914+0000 7f57e7000700 1 --2- 172.21.15.27:0/406332860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e0102900 0x7f57e010f4c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f57d0012de0 tx=0x7f57d0009840 comp rx=0 tx=0).stop 2024-04-16T05:32:05.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.915+0000 7f57e7000700 1 -- 172.21.15.27:0/406332860 shutdown_connections 2024-04-16T05:32:05.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.915+0000 7f57e7000700 1 --2- 172.21.15.27:0/406332860 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f57c403acd0 0x7f57c403d180 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:05.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.915+0000 7f57e7000700 1 --2- 172.21.15.27:0/406332860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f57e0102900 0x7f57e010f4c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:05.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.915+0000 7f57e7000700 1 -- 172.21.15.27:0/406332860 >> 172.21.15.27:0/406332860 conn(0x7f57e00fe0a0 msgr2=0x7f57e00feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:05.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.915+0000 7f57e7000700 1 -- 172.21.15.27:0/406332860 shutdown_connections 2024-04-16T05:32:05.625 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:04.915+0000 7f57e7000700 1 -- 172.21.15.27:0/406332860 wait complete. 2024-04-16T05:32:06.492 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:06 smithi027 ceph-mon[29206]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:32:07.133 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.229+0000 7f09d2800700 1 Processor -- start 2024-04-16T05:32:07.133 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.230+0000 7f09d2800700 1 -- start start 2024-04-16T05:32:07.133 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.230+0000 7f09d2800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc077c10 0x7f09cc076150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:07.133 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.230+0000 7f09d2800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc078020 con 0x7f09cc077c10 2024-04-16T05:32:07.133 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.231+0000 7f09d1400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc077c10 0x7f09cc076150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:07.133 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.231+0000 7f09d1400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc077c10 0x7f09cc076150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40766/0 (socket says 172.21.15.27:40766) 2024-04-16T05:32:07.133 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.231+0000 7f09d1400700 1 -- 172.21.15.27:0/2117296087 learned_addr learned my addr 172.21.15.27:0/2117296087 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:07.134 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.231+0000 7f09d1400700 1 -- 172.21.15.27:0/2117296087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09cc076690 con 0x7f09cc077c10 2024-04-16T05:32:07.134 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.232+0000 7f09d1400700 1 --2- 172.21.15.27:0/2117296087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc077c10 0x7f09cc076150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009e50 tx=0x7f09c0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b336c95ad6f8982 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:07.134 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.232+0000 7f09cbe00700 1 -- 172.21.15.27:0/2117296087 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09c00092e0 con 0x7f09cc077c10 2024-04-16T05:32:07.134 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.232+0000 7f09cbe00700 1 -- 172.21.15.27:0/2117296087 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09c0012430 con 0x7f09cc077c10 2024-04-16T05:32:07.134 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.232+0000 7f09cbe00700 1 -- 172.21.15.27:0/2117296087 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09c00075a0 con 0x7f09cc077c10 2024-04-16T05:32:07.137 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.232+0000 7f09d2800700 1 -- 172.21.15.27:0/2117296087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc077c10 msgr2=0x7f09cc076150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:07.137 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.232+0000 7f09d2800700 1 --2- 172.21.15.27:0/2117296087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc077c10 0x7f09cc076150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009e50 tx=0x7f09c0009840 comp rx=0 tx=0).stop 2024-04-16T05:32:07.138 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.233+0000 7f09d2800700 1 -- 172.21.15.27:0/2117296087 shutdown_connections 2024-04-16T05:32:07.138 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.233+0000 7f09d2800700 1 --2- 172.21.15.27:0/2117296087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc077c10 0x7f09cc076150 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:07.138 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.233+0000 7f09d2800700 1 -- 172.21.15.27:0/2117296087 >> 172.21.15.27:0/2117296087 conn(0x7f09cc0fde30 msgr2=0x7f09cc100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:07.138 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.233+0000 7f09d2800700 1 -- 172.21.15.27:0/2117296087 shutdown_connections 2024-04-16T05:32:07.138 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.233+0000 7f09d2800700 1 -- 172.21.15.27:0/2117296087 wait complete. 2024-04-16T05:32:07.138 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.234+0000 7f09d2800700 1 Processor -- start 2024-04-16T05:32:07.138 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.234+0000 7f09d2800700 1 -- start start 2024-04-16T05:32:07.138 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.234+0000 7f09d2800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc077c10 0x7f09cc113700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:07.138 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.234+0000 7f09d2800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc113c40 con 0x7f09cc077c10 2024-04-16T05:32:07.138 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.234+0000 7f09d1400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc077c10 0x7f09cc113700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:07.138 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.234+0000 7f09d1400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc077c10 0x7f09cc113700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40776/0 (socket says 172.21.15.27:40776) 2024-04-16T05:32:07.138 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.234+0000 7f09d1400700 1 -- 172.21.15.27:0/1049491289 learned_addr learned my addr 172.21.15.27:0/1049491289 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:07.138 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.235+0000 7f09d1400700 1 -- 172.21.15.27:0/1049491289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c0009ae0 con 0x7f09cc077c10 2024-04-16T05:32:07.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.235+0000 7f09d1400700 1 --2- 172.21.15.27:0/1049491289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc077c10 0x7f09cc113700 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009fd0 tx=0x7f09c0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:07.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.235+0000 7f09c9600700 1 -- 172.21.15.27:0/1049491289 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09c00092e0 con 0x7f09cc077c10 2024-04-16T05:32:07.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.235+0000 7f09c9600700 1 -- 172.21.15.27:0/1049491289 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09c0007ce0 con 0x7f09cc077c10 2024-04-16T05:32:07.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.235+0000 7f09c9600700 1 -- 172.21.15.27:0/1049491289 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09c0025960 con 0x7f09cc077c10 2024-04-16T05:32:07.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.235+0000 7f09d2800700 1 -- 172.21.15.27:0/1049491289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09cc113e40 con 0x7f09cc077c10 2024-04-16T05:32:07.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.236+0000 7f09d2800700 1 -- 172.21.15.27:0/1049491289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09cc1149b0 con 0x7f09cc077c10 2024-04-16T05:32:07.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.237+0000 7f09d2800700 1 -- 172.21.15.27:0/1049491289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09cc05f5f0 con 0x7f09cc077c10 2024-04-16T05:32:07.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.237+0000 7f09c9600700 1 -- 172.21.15.27:0/1049491289 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7f09c0019050 con 0x7f09cc077c10 2024-04-16T05:32:07.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.238+0000 7f09c9600700 1 --2- 172.21.15.27:0/1049491289 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f09b4036850 0x7f09b4038d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:07.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.238+0000 7f09c9600700 1 -- 172.21.15.27:0/1049491289 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f09c004a050 con 0x7f09cc077c10 2024-04-16T05:32:07.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.238+0000 7f09d0a00700 1 --2- 172.21.15.27:0/1049491289 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f09b4036850 0x7f09b4038d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:07.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.239+0000 7f09d0a00700 1 --2- 172.21.15.27:0/1049491289 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f09b4036850 0x7f09b4038d00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f09b8009990 tx=0x7f09b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:07.139 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.243+0000 7f09c9600700 1 -- 172.21.15.27:0/1049491289 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f09c0011390 con 0x7f09cc077c10 2024-04-16T05:32:07.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.462+0000 7f09d2800700 1 -- 172.21.15.27:0/1049491289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f09cc04bed0 con 0x7f09cc077c10 2024-04-16T05:32:07.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.470+0000 7f09c9600700 1 -- 172.21.15.27:0/1049491289 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f09c0020070 con 0x7f09cc077c10 2024-04-16T05:32:07.140 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.473+0000 7f09d2800700 1 -- 172.21.15.27:0/1049491289 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f09b4036850 msgr2=0x7f09b4038d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:07.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.473+0000 7f09d2800700 1 --2- 172.21.15.27:0/1049491289 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f09b4036850 0x7f09b4038d00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f09b8009990 tx=0x7f09b8006e40 comp rx=0 tx=0).stop 2024-04-16T05:32:07.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.473+0000 7f09d2800700 1 -- 172.21.15.27:0/1049491289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc077c10 msgr2=0x7f09cc113700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:07.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.473+0000 7f09d2800700 1 --2- 172.21.15.27:0/1049491289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc077c10 0x7f09cc113700 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009fd0 tx=0x7f09c0009840 comp rx=0 tx=0).stop 2024-04-16T05:32:07.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.474+0000 7f09d2800700 1 -- 172.21.15.27:0/1049491289 shutdown_connections 2024-04-16T05:32:07.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.474+0000 7f09d2800700 1 --2- 172.21.15.27:0/1049491289 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f09b4036850 0x7f09b4038d00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:07.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.474+0000 7f09d2800700 1 --2- 172.21.15.27:0/1049491289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09cc077c10 0x7f09cc113700 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:07.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.474+0000 7f09d2800700 1 -- 172.21.15.27:0/1049491289 >> 172.21.15.27:0/1049491289 conn(0x7f09cc0fde30 msgr2=0x7f09cc0fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:07.141 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.474+0000 7f09d2800700 1 -- 172.21.15.27:0/1049491289 shutdown_connections 2024-04-16T05:32:07.142 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:06.474+0000 7f09d2800700 1 -- 172.21.15.27:0/1049491289 wait complete. 2024-04-16T05:32:07.142 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling the dashboard module... 2024-04-16T05:32:07.734 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:07 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1049491289' entity='client.admin' 2024-04-16T05:32:07.734 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:07 smithi027 ceph-mon[29206]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:32:08.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:08 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/3008666857' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-16T05:32:09.249 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.734+0000 7efd6da00700 1 Processor -- start 2024-04-16T05:32:09.250 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.735+0000 7efd6da00700 1 -- start start 2024-04-16T05:32:09.251 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.735+0000 7efd6da00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd68077b50 0x7efd68076150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:09.251 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.735+0000 7efd6da00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd68077f60 con 0x7efd68077b50 2024-04-16T05:32:09.251 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.736+0000 7efd66a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd68077b50 0x7efd68076150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:09.251 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.736+0000 7efd66a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd68077b50 0x7efd68076150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40788/0 (socket says 172.21.15.27:40788) 2024-04-16T05:32:09.251 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.736+0000 7efd66a00700 1 -- 172.21.15.27:0/414460837 learned_addr learned my addr 172.21.15.27:0/414460837 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:09.251 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.736+0000 7efd66a00700 1 -- 172.21.15.27:0/414460837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd68076690 con 0x7efd68077b50 2024-04-16T05:32:09.251 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.737+0000 7efd66a00700 1 --2- 172.21.15.27:0/414460837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd68077b50 0x7efd68076150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efd580098e0 tx=0x7efd58009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a82bd0f18898b400 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:09.251 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.737+0000 7efd65600700 1 -- 172.21.15.27:0/414460837 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd5800b860 con 0x7efd68077b50 2024-04-16T05:32:09.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.737+0000 7efd65600700 1 -- 172.21.15.27:0/414460837 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd5800be90 con 0x7efd68077b50 2024-04-16T05:32:09.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.737+0000 7efd65600700 1 -- 172.21.15.27:0/414460837 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd580103c0 con 0x7efd68077b50 2024-04-16T05:32:09.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.738+0000 7efd6da00700 1 -- 172.21.15.27:0/414460837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd68077b50 msgr2=0x7efd68076150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:09.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.738+0000 7efd6da00700 1 --2- 172.21.15.27:0/414460837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd68077b50 0x7efd68076150 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efd580098e0 tx=0x7efd58009250 comp rx=0 tx=0).stop 2024-04-16T05:32:09.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.738+0000 7efd6da00700 1 -- 172.21.15.27:0/414460837 shutdown_connections 2024-04-16T05:32:09.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.738+0000 7efd6da00700 1 --2- 172.21.15.27:0/414460837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd68077b50 0x7efd68076150 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:09.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.738+0000 7efd6da00700 1 -- 172.21.15.27:0/414460837 >> 172.21.15.27:0/414460837 conn(0x7efd680fe110 msgr2=0x7efd68100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:09.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.738+0000 7efd6da00700 1 -- 172.21.15.27:0/414460837 shutdown_connections 2024-04-16T05:32:09.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.738+0000 7efd6da00700 1 -- 172.21.15.27:0/414460837 wait complete. 2024-04-16T05:32:09.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.740+0000 7efd6da00700 1 Processor -- start 2024-04-16T05:32:09.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.740+0000 7efd6da00700 1 -- start start 2024-04-16T05:32:09.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.740+0000 7efd6da00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd68077b50 0x7efd681138a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:09.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.740+0000 7efd6da00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd68113de0 con 0x7efd68077b50 2024-04-16T05:32:09.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.740+0000 7efd66a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd68077b50 0x7efd681138a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:09.254 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.740+0000 7efd66a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd68077b50 0x7efd681138a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40798/0 (socket says 172.21.15.27:40798) 2024-04-16T05:32:09.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.740+0000 7efd66a00700 1 -- 172.21.15.27:0/3008666857 learned_addr learned my addr 172.21.15.27:0/3008666857 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:09.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.741+0000 7efd66a00700 1 -- 172.21.15.27:0/3008666857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd580094f0 con 0x7efd68077b50 2024-04-16T05:32:09.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.741+0000 7efd66a00700 1 --2- 172.21.15.27:0/3008666857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd68077b50 0x7efd681138a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efd58011c10 tx=0x7efd58009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:09.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.741+0000 7efd5ea00700 1 -- 172.21.15.27:0/3008666857 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd58010630 con 0x7efd68077b50 2024-04-16T05:32:09.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.742+0000 7efd5ea00700 1 -- 172.21.15.27:0/3008666857 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd58010c60 con 0x7efd68077b50 2024-04-16T05:32:09.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.742+0000 7efd6da00700 1 -- 172.21.15.27:0/3008666857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd68113fe0 con 0x7efd68077b50 2024-04-16T05:32:09.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.742+0000 7efd5ea00700 1 -- 172.21.15.27:0/3008666857 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd5801ad10 con 0x7efd68077b50 2024-04-16T05:32:09.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.742+0000 7efd6da00700 1 -- 172.21.15.27:0/3008666857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd68114b50 con 0x7efd68077b50 2024-04-16T05:32:09.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.743+0000 7efd6da00700 1 -- 172.21.15.27:0/3008666857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd6805f5f0 con 0x7efd68077b50 2024-04-16T05:32:09.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.743+0000 7efd5ea00700 1 -- 172.21.15.27:0/3008666857 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) v1 ==== 43376+0+0 (secure 0 0 0) 0x7efd5801a5f0 con 0x7efd68077b50 2024-04-16T05:32:09.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.743+0000 7efd5ea00700 1 --2- 172.21.15.27:0/3008666857 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7efd4c0368a0 0x7efd4c038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:09.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.743+0000 7efd5ea00700 1 -- 172.21.15.27:0/3008666857 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7efd58049f70 con 0x7efd68077b50 2024-04-16T05:32:09.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.744+0000 7efd66000700 1 --2- 172.21.15.27:0/3008666857 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7efd4c0368a0 0x7efd4c038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:09.255 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.744+0000 7efd66000700 1 --2- 172.21.15.27:0/3008666857 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7efd4c0368a0 0x7efd4c038d50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efd54009990 tx=0x7efd54006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:09.256 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:07.749+0000 7efd5ea00700 1 -- 172.21.15.27:0/3008666857 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7efd58024590 con 0x7efd68077b50 2024-04-16T05:32:09.256 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:08.021+0000 7efd6da00700 1 -- 172.21.15.27:0/3008666857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7efd6804bed0 con 0x7efd68077b50 2024-04-16T05:32:09.256 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:08.484+0000 7efd5ea00700 1 -- 172.21.15.27:0/3008666857 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 9) v1 ==== 43389+0+0 (secure 0 0 0) 0x7efd5802d350 con 0x7efd68077b50 2024-04-16T05:32:09.256 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:08.486+0000 7efd5ea00700 1 -- 172.21.15.27:0/3008666857 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7efd58010dd0 con 0x7efd68077b50 2024-04-16T05:32:09.257 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:08.490+0000 7efd6da00700 1 -- 172.21.15.27:0/3008666857 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7efd4c0368a0 msgr2=0x7efd4c038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:09.257 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:08.490+0000 7efd6da00700 1 --2- 172.21.15.27:0/3008666857 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7efd4c0368a0 0x7efd4c038d50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efd54009990 tx=0x7efd54006e40 comp rx=0 tx=0).stop 2024-04-16T05:32:09.257 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:08.490+0000 7efd6da00700 1 -- 172.21.15.27:0/3008666857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd68077b50 msgr2=0x7efd681138a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:09.257 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:08.490+0000 7efd6da00700 1 --2- 172.21.15.27:0/3008666857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd68077b50 0x7efd681138a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efd58011c10 tx=0x7efd58009250 comp rx=0 tx=0).stop 2024-04-16T05:32:09.257 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:08.490+0000 7efd6da00700 1 -- 172.21.15.27:0/3008666857 shutdown_connections 2024-04-16T05:32:09.257 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:08.490+0000 7efd6da00700 1 --2- 172.21.15.27:0/3008666857 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7efd4c0368a0 0x7efd4c038d50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:09.257 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:08.490+0000 7efd6da00700 1 --2- 172.21.15.27:0/3008666857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efd68077b50 0x7efd681138a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:09.257 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:08.490+0000 7efd6da00700 1 -- 172.21.15.27:0/3008666857 >> 172.21.15.27:0/3008666857 conn(0x7efd680fe110 msgr2=0x7efd680febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:09.257 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:08.490+0000 7efd6da00700 1 -- 172.21.15.27:0/3008666857 shutdown_connections 2024-04-16T05:32:09.257 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:08.490+0000 7efd6da00700 1 -- 172.21.15.27:0/3008666857 wait complete. 2024-04-16T05:32:09.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:09 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/3008666857' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-16T05:32:09.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:09 smithi027 ceph-mon[29206]: mgrmap e9: smithi027.qgzypi(active, since 33s) 2024-04-16T05:32:10.784 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:10 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/3727394253' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-16T05:32:10.905 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-16T05:32:10.905 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-16T05:32:10.905 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-16T05:32:10.905 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "active_name": "smithi027.qgzypi", 2024-04-16T05:32:10.905 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-16T05:32:10.906 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-16T05:32:10.906 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.866+0000 7f5d59800700 1 Processor -- start 2024-04-16T05:32:10.906 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.867+0000 7f5d59800700 1 -- start start 2024-04-16T05:32:10.906 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.867+0000 7f5d59800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d54102900 0x7f5d54102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:10.906 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.867+0000 7f5d59800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d54103250 con 0x7f5d54102900 2024-04-16T05:32:10.906 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.867+0000 7f5d52a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d54102900 0x7f5d54102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:10.906 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.868+0000 7f5d52a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d54102900 0x7f5d54102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40820/0 (socket says 172.21.15.27:40820) 2024-04-16T05:32:10.906 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.868+0000 7f5d52a00700 1 -- 172.21.15.27:0/1521951772 learned_addr learned my addr 172.21.15.27:0/1521951772 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:10.906 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.868+0000 7f5d52a00700 1 -- 172.21.15.27:0/1521951772 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d54103390 con 0x7f5d54102900 2024-04-16T05:32:10.906 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.868+0000 7f5d52a00700 1 --2- 172.21.15.27:0/1521951772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d54102900 0x7f5d54102d10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5d440098e0 tx=0x7f5d44009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf060ef9f16da71c server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:10.906 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.869+0000 7f5d51600700 1 -- 172.21.15.27:0/1521951772 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d4400b860 con 0x7f5d54102900 2024-04-16T05:32:10.906 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.869+0000 7f5d51600700 1 -- 172.21.15.27:0/1521951772 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d4400be90 con 0x7f5d54102900 2024-04-16T05:32:10.906 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.869+0000 7f5d51600700 1 -- 172.21.15.27:0/1521951772 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d440103c0 con 0x7f5d54102900 2024-04-16T05:32:10.907 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.870+0000 7f5d59800700 1 -- 172.21.15.27:0/1521951772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d54102900 msgr2=0x7f5d54102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:10.907 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.870+0000 7f5d59800700 1 --2- 172.21.15.27:0/1521951772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d54102900 0x7f5d54102d10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5d440098e0 tx=0x7f5d44009250 comp rx=0 tx=0).stop 2024-04-16T05:32:10.907 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.870+0000 7f5d59800700 1 -- 172.21.15.27:0/1521951772 shutdown_connections 2024-04-16T05:32:10.907 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.870+0000 7f5d59800700 1 --2- 172.21.15.27:0/1521951772 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d54102900 0x7f5d54102d10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:10.907 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.870+0000 7f5d59800700 1 -- 172.21.15.27:0/1521951772 >> 172.21.15.27:0/1521951772 conn(0x7f5d540fe0a0 msgr2=0x7f5d541004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:10.907 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.870+0000 7f5d59800700 1 -- 172.21.15.27:0/1521951772 shutdown_connections 2024-04-16T05:32:10.907 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.870+0000 7f5d59800700 1 -- 172.21.15.27:0/1521951772 wait complete. 2024-04-16T05:32:10.907 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.871+0000 7f5d59800700 1 Processor -- start 2024-04-16T05:32:10.907 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.871+0000 7f5d59800700 1 -- start start 2024-04-16T05:32:10.907 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.871+0000 7f5d59800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d54102900 0x7f5d54113850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:10.907 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.871+0000 7f5d59800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d54113d90 con 0x7f5d54102900 2024-04-16T05:32:10.907 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.872+0000 7f5d52a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d54102900 0x7f5d54113850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:10.907 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.872+0000 7f5d52a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d54102900 0x7f5d54113850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40836/0 (socket says 172.21.15.27:40836) 2024-04-16T05:32:10.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.872+0000 7f5d52a00700 1 -- 172.21.15.27:0/3727394253 learned_addr learned my addr 172.21.15.27:0/3727394253 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:10.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.872+0000 7f5d52a00700 1 -- 172.21.15.27:0/3727394253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d440094f0 con 0x7f5d54102900 2024-04-16T05:32:10.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.872+0000 7f5d52a00700 1 --2- 172.21.15.27:0/3727394253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d54102900 0x7f5d54113850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5d44006b00 tx=0x7f5d44009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:10.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.873+0000 7f5d4aa00700 1 -- 172.21.15.27:0/3727394253 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d44010840 con 0x7f5d54102900 2024-04-16T05:32:10.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.873+0000 7f5d4aa00700 1 -- 172.21.15.27:0/3727394253 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d44010e70 con 0x7f5d54102900 2024-04-16T05:32:10.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.873+0000 7f5d59800700 1 -- 172.21.15.27:0/3727394253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d54113f90 con 0x7f5d54102900 2024-04-16T05:32:10.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.873+0000 7f5d59800700 1 -- 172.21.15.27:0/3727394253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d54114b00 con 0x7f5d54102900 2024-04-16T05:32:10.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.873+0000 7f5d4aa00700 1 -- 172.21.15.27:0/3727394253 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d4401ae70 con 0x7f5d54102900 2024-04-16T05:32:10.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.874+0000 7f5d4aa00700 1 -- 172.21.15.27:0/3727394253 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) v1 ==== 43389+0+0 (secure 0 0 0) 0x7f5d440244a0 con 0x7f5d54102900 2024-04-16T05:32:10.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.874+0000 7f5d3fe00700 1 -- 172.21.15.27:0/3727394253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d2c0052b0 con 0x7f5d54102900 2024-04-16T05:32:10.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.875+0000 7f5d4aa00700 1 --2- 172.21.15.27:0/3727394253 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f5d380368f0 0x7f5d38038da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:10.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.875+0000 7f5d4aa00700 1 -- 172.21.15.27:0/3727394253 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5d4404abf0 con 0x7f5d54102900 2024-04-16T05:32:10.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.875+0000 7f5d52000700 1 -- 172.21.15.27:0/3727394253 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f5d380368f0 msgr2=0x7f5d38038da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2779081505 2024-04-16T05:32:10.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.875+0000 7f5d52000700 1 --2- 172.21.15.27:0/3727394253 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f5d380368f0 0x7f5d38038da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T05:32:10.908 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:09.880+0000 7f5d4aa00700 1 -- 172.21.15.27:0/3727394253 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f5d44023b80 con 0x7f5d54102900 2024-04-16T05:32:10.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:10.076+0000 7f5d52000700 1 -- 172.21.15.27:0/3727394253 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f5d380368f0 msgr2=0x7f5d38038da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2779081505 2024-04-16T05:32:10.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:10.076+0000 7f5d52000700 1 --2- 172.21.15.27:0/3727394253 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f5d380368f0 0x7f5d38038da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T05:32:10.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:10.152+0000 7f5d3fe00700 1 -- 172.21.15.27:0/3727394253 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f5d2c005d70 con 0x7f5d54102900 2024-04-16T05:32:10.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:10.153+0000 7f5d4aa00700 1 -- 172.21.15.27:0/3727394253 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f5d44023b80 con 0x7f5d54102900 2024-04-16T05:32:10.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:10.155+0000 7f5d3fe00700 1 -- 172.21.15.27:0/3727394253 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f5d380368f0 msgr2=0x7f5d38038da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T05:32:10.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:10.155+0000 7f5d3fe00700 1 --2- 172.21.15.27:0/3727394253 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f5d380368f0 0x7f5d38038da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:10.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:10.155+0000 7f5d3fe00700 1 -- 172.21.15.27:0/3727394253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d54102900 msgr2=0x7f5d54113850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:10.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:10.155+0000 7f5d3fe00700 1 --2- 172.21.15.27:0/3727394253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d54102900 0x7f5d54113850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5d44006b00 tx=0x7f5d44009250 comp rx=0 tx=0).stop 2024-04-16T05:32:10.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:10.155+0000 7f5d3fe00700 1 -- 172.21.15.27:0/3727394253 shutdown_connections 2024-04-16T05:32:10.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:10.155+0000 7f5d3fe00700 1 --2- 172.21.15.27:0/3727394253 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f5d380368f0 0x7f5d38038da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:10.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:10.155+0000 7f5d3fe00700 1 --2- 172.21.15.27:0/3727394253 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5d54102900 0x7f5d54113850 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:10.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:10.155+0000 7f5d3fe00700 1 -- 172.21.15.27:0/3727394253 >> 172.21.15.27:0/3727394253 conn(0x7f5d540fe0a0 msgr2=0x7f5d540fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:10.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:10.156+0000 7f5d3fe00700 1 -- 172.21.15.27:0/3727394253 shutdown_connections 2024-04-16T05:32:10.909 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:10.156+0000 7f5d3fe00700 1 -- 172.21.15.27:0/3727394253 wait complete. 2024-04-16T05:32:10.909 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for the mgr to restart... 2024-04-16T05:32:10.910 INFO:teuthology.orchestra.run.smithi027.stdout:Waiting for mgr epoch 9... 2024-04-16T05:32:17.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:17 smithi027 ceph-mon[29206]: Active manager daemon smithi027.qgzypi restarted 2024-04-16T05:32:17.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:17 smithi027 ceph-mon[29206]: Activating manager daemon smithi027.qgzypi 2024-04-16T05:32:17.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:17 smithi027 ceph-mon[29206]: osdmap e3: 0 total, 0 up, 0 in 2024-04-16T05:32:17.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:17 smithi027 ceph-mon[29206]: mgrmap e10: smithi027.qgzypi(active, starting, since 0.0113194s) 2024-04-16T05:32:17.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:17 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-16T05:32:17.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:17 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr metadata", "who": "smithi027.qgzypi", "id": "smithi027.qgzypi"}]: dispatch 2024-04-16T05:32:17.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:17 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T05:32:17.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:17 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T05:32:17.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:17 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T05:32:17.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:17 smithi027 ceph-mon[29206]: Manager daemon smithi027.qgzypi is now available 2024-04-16T05:32:18.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:18 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:32:18.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:18 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:32:18.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:18 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.qgzypi/mirror_snapshot_schedule"}]: dispatch 2024-04-16T05:32:18.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:18 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.qgzypi/trash_purge_schedule"}]: dispatch 2024-04-16T05:32:18.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:18 smithi027 ceph-mon[29206]: mgrmap e11: smithi027.qgzypi(active, since 1.01249s) 2024-04-16T05:32:19.776 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout { 2024-04-16T05:32:19.776 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-16T05:32:19.776 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-16T05:32:19.776 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout } 2024-04-16T05:32:19.776 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.488+0000 7f4eeb600700 1 Processor -- start 2024-04-16T05:32:19.776 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.488+0000 7f4eeb600700 1 -- start start 2024-04-16T05:32:19.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.488+0000 7f4eeb600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ee4077c00 0x7f4ee40761b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:19.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.488+0000 7f4eeb600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee4078010 con 0x7f4ee4077c00 2024-04-16T05:32:19.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.489+0000 7f4ee8c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ee4077c00 0x7f4ee40761b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:19.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.489+0000 7f4ee8c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ee4077c00 0x7f4ee40761b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40838/0 (socket says 172.21.15.27:40838) 2024-04-16T05:32:19.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.489+0000 7f4ee8c00700 1 -- 172.21.15.27:0/1061854767 learned_addr learned my addr 172.21.15.27:0/1061854767 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:19.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.489+0000 7f4ee8c00700 1 -- 172.21.15.27:0/1061854767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee40766f0 con 0x7f4ee4077c00 2024-04-16T05:32:19.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.490+0000 7f4ee8c00700 1 --2- 172.21.15.27:0/1061854767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ee4077c00 0x7f4ee40761b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4ed80098e0 tx=0x7f4ed8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d12e96d94f14ae server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:19.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.490+0000 7f4ee3400700 1 -- 172.21.15.27:0/1061854767 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ed800b860 con 0x7f4ee4077c00 2024-04-16T05:32:19.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.490+0000 7f4ee3400700 1 -- 172.21.15.27:0/1061854767 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ed800be90 con 0x7f4ee4077c00 2024-04-16T05:32:19.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.490+0000 7f4ee3400700 1 -- 172.21.15.27:0/1061854767 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ed80103c0 con 0x7f4ee4077c00 2024-04-16T05:32:19.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.491+0000 7f4eeb600700 1 -- 172.21.15.27:0/1061854767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ee4077c00 msgr2=0x7f4ee40761b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:19.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.491+0000 7f4eeb600700 1 --2- 172.21.15.27:0/1061854767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ee4077c00 0x7f4ee40761b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4ed80098e0 tx=0x7f4ed8009250 comp rx=0 tx=0).stop 2024-04-16T05:32:19.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.491+0000 7f4eeb600700 1 -- 172.21.15.27:0/1061854767 shutdown_connections 2024-04-16T05:32:19.777 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.491+0000 7f4eeb600700 1 --2- 172.21.15.27:0/1061854767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ee4077c00 0x7f4ee40761b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:19.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.491+0000 7f4eeb600700 1 -- 172.21.15.27:0/1061854767 >> 172.21.15.27:0/1061854767 conn(0x7f4ee40fe0a0 msgr2=0x7f4ee41004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:19.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.491+0000 7f4eeb600700 1 -- 172.21.15.27:0/1061854767 shutdown_connections 2024-04-16T05:32:19.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.491+0000 7f4eeb600700 1 -- 172.21.15.27:0/1061854767 wait complete. 2024-04-16T05:32:19.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.492+0000 7f4eeb600700 1 Processor -- start 2024-04-16T05:32:19.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.492+0000 7f4eeb600700 1 -- start start 2024-04-16T05:32:19.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.492+0000 7f4eeb600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ee4077c00 0x7f4ee4113880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:19.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.492+0000 7f4eeb600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee4113dc0 con 0x7f4ee4077c00 2024-04-16T05:32:19.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.492+0000 7f4ee8c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ee4077c00 0x7f4ee4113880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:19.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.493+0000 7f4ee8c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ee4077c00 0x7f4ee4113880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40844/0 (socket says 172.21.15.27:40844) 2024-04-16T05:32:19.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.493+0000 7f4ee8c00700 1 -- 172.21.15.27:0/3790077626 learned_addr learned my addr 172.21.15.27:0/3790077626 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:19.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.493+0000 7f4ee8c00700 1 -- 172.21.15.27:0/3790077626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed80094f0 con 0x7f4ee4077c00 2024-04-16T05:32:19.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.493+0000 7f4ee8c00700 1 --2- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ee4077c00 0x7f4ee4113880 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4ed80105b0 tx=0x7f4ed8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:19.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.493+0000 7f4ee0c00700 1 -- 172.21.15.27:0/3790077626 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ed8010840 con 0x7f4ee4077c00 2024-04-16T05:32:19.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.494+0000 7f4ee0c00700 1 -- 172.21.15.27:0/3790077626 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ed8010e70 con 0x7f4ee4077c00 2024-04-16T05:32:19.778 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.494+0000 7f4ee0c00700 1 -- 172.21.15.27:0/3790077626 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ed801ae30 con 0x7f4ee4077c00 2024-04-16T05:32:19.779 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.494+0000 7f4eeb600700 1 -- 172.21.15.27:0/3790077626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee4113fc0 con 0x7f4ee4077c00 2024-04-16T05:32:19.779 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.494+0000 7f4eeb600700 1 -- 172.21.15.27:0/3790077626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee4114b30 con 0x7f4ee4077c00 2024-04-16T05:32:19.779 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.495+0000 7f4ee0c00700 1 -- 172.21.15.27:0/3790077626 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) v1 ==== 43389+0+0 (secure 0 0 0) 0x7f4ed801a430 con 0x7f4ee4077c00 2024-04-16T05:32:19.779 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.495+0000 7f4ee0c00700 1 --2- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f4ecc0368a0 0x7f4ecc038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:19.779 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.495+0000 7f4ee0c00700 1 -- 172.21.15.27:0/3790077626 --> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ecc039460 con 0x7f4ecc0368a0 2024-04-16T05:32:19.779 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.495+0000 7f4ee0c00700 1 -- 172.21.15.27:0/3790077626 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f4ed804a2e0 con 0x7f4ee4077c00 2024-04-16T05:32:19.779 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.496+0000 7f4ee3e00700 1 -- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f4ecc0368a0 msgr2=0x7f4ecc038d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2779081505 2024-04-16T05:32:19.779 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.496+0000 7f4ee3e00700 1 --2- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f4ecc0368a0 0x7f4ecc038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T05:32:19.779 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.697+0000 7f4ee3e00700 1 -- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f4ecc0368a0 msgr2=0x7f4ecc038d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2779081505 2024-04-16T05:32:19.779 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:11.697+0000 7f4ee3e00700 1 --2- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f4ecc0368a0 0x7f4ecc038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T05:32:19.779 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:12.098+0000 7f4ee3e00700 1 -- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f4ecc0368a0 msgr2=0x7f4ecc038d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2779081505 2024-04-16T05:32:19.779 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:12.098+0000 7f4ee3e00700 1 --2- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f4ecc0368a0 0x7f4ecc038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T05:32:19.780 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:12.899+0000 7f4ee3e00700 1 -- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f4ecc0368a0 msgr2=0x7f4ecc038d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2779081505 2024-04-16T05:32:19.780 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:12.899+0000 7f4ee3e00700 1 --2- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f4ecc0368a0 0x7f4ecc038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T05:32:19.780 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:14.501+0000 7f4ee3e00700 1 -- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f4ecc0368a0 msgr2=0x7f4ecc038d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2779081505 2024-04-16T05:32:19.780 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:14.501+0000 7f4ee3e00700 1 --2- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f4ecc0368a0 0x7f4ecc038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T05:32:19.780 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:17.537+0000 7f4ee0c00700 1 -- 172.21.15.27:0/3790077626 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mgrmap(e 10) v1 ==== 43156+0+0 (secure 0 0 0) 0x7f4ed801a710 con 0x7f4ee4077c00 2024-04-16T05:32:19.780 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:17.537+0000 7f4ee0c00700 1 -- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f4ecc0368a0 msgr2=0x7f4ecc038d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T05:32:19.780 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:17.537+0000 7f4ee0c00700 1 --2- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f4ecc0368a0 0x7f4ecc038d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:19.780 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:18.536+0000 7f4ee0c00700 1 -- 172.21.15.27:0/3790077626 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 11) v1 ==== 43283+0+0 (secure 0 0 0) 0x7f4ed8019d90 con 0x7f4ee4077c00 2024-04-16T05:32:19.780 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:18.537+0000 7f4ee0c00700 1 --2- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f4ecc03a1d0 0x7f4ecc03c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:19.780 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:18.537+0000 7f4ee0c00700 1 -- 172.21.15.27:0/3790077626 --> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ecc039460 con 0x7f4ecc03a1d0 2024-04-16T05:32:19.780 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:18.539+0000 7f4ee3e00700 1 --2- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f4ecc03a1d0 0x7f4ecc03c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:19.780 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:18.540+0000 7f4ee3e00700 1 --2- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f4ecc03a1d0 0x7f4ecc03c5b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4ed00039d0 tx=0x7f4ed00092f0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:19.780 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:18.541+0000 7f4ee0c00700 1 -- 172.21.15.27:0/3790077626 <== mgr.14162 v2:172.21.15.27:6800/2700787222 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f4ecc039460 con 0x7f4ecc03a1d0 2024-04-16T05:32:19.780 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:18.550+0000 7f4eeb600700 1 -- 172.21.15.27:0/3790077626 --> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f4ee4114450 con 0x7f4ecc03a1d0 2024-04-16T05:32:19.780 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:18.551+0000 7f4ee0c00700 1 -- 172.21.15.27:0/3790077626 <== mgr.14162 v2:172.21.15.27:6800/2700787222 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f4ee4114450 con 0x7f4ecc03a1d0 2024-04-16T05:32:19.781 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:18.551+0000 7f4eeb600700 1 -- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f4ecc03a1d0 msgr2=0x7f4ecc03c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:19.781 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:18.551+0000 7f4eeb600700 1 --2- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f4ecc03a1d0 0x7f4ecc03c5b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4ed00039d0 tx=0x7f4ed00092f0 comp rx=0 tx=0).stop 2024-04-16T05:32:19.781 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:18.551+0000 7f4eeb600700 1 -- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ee4077c00 msgr2=0x7f4ee4113880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:19.781 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:18.551+0000 7f4eeb600700 1 --2- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ee4077c00 0x7f4ee4113880 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4ed80105b0 tx=0x7f4ed8009250 comp rx=0 tx=0).stop 2024-04-16T05:32:19.781 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:18.551+0000 7f4eeb600700 1 -- 172.21.15.27:0/3790077626 shutdown_connections 2024-04-16T05:32:19.781 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:18.551+0000 7f4eeb600700 1 --2- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f4ecc03a1d0 0x7f4ecc03c5b0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:19.782 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:18.551+0000 7f4eeb600700 1 --2- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ee4077c00 0x7f4ee4113880 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:19.782 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:18.551+0000 7f4eeb600700 1 --2- 172.21.15.27:0/3790077626 >> [v2:172.21.15.27:6800/2779081505,v1:172.21.15.27:6801/2779081505] conn(0x7f4ecc0368a0 0x7f4ecc038d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:19.782 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:18.551+0000 7f4eeb600700 1 -- 172.21.15.27:0/3790077626 >> 172.21.15.27:0/3790077626 conn(0x7f4ee40fe0a0 msgr2=0x7f4ee40feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:19.782 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:18.552+0000 7f4eeb600700 1 -- 172.21.15.27:0/3790077626 shutdown_connections 2024-04-16T05:32:19.782 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:18.552+0000 7f4eeb600700 1 -- 172.21.15.27:0/3790077626 wait complete. 2024-04-16T05:32:19.782 INFO:teuthology.orchestra.run.smithi027.stdout:mgr epoch 9 is available 2024-04-16T05:32:19.782 INFO:teuthology.orchestra.run.smithi027.stdout:Generating a dashboard self-signed certificate... 2024-04-16T05:32:19.927 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:19 smithi027 ceph-mon[29206]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-16T05:32:19.928 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:19 smithi027 ceph-mon[29206]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-16T05:32:19.928 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:19 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:20.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:20 smithi027 ceph-mon[29206]: [16/Apr/2024:05:32:18] ENGINE Bus STARTING 2024-04-16T05:32:20.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:20 smithi027 ceph-mon[29206]: [16/Apr/2024:05:32:18] ENGINE Serving on https://172.21.15.27:7150 2024-04-16T05:32:20.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:20 smithi027 ceph-mon[29206]: [16/Apr/2024:05:32:18] ENGINE Bus STARTED 2024-04-16T05:32:20.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:20 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:20.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:20 smithi027 ceph-mon[29206]: mgrmap e12: smithi027.qgzypi(active, since 2s) 2024-04-16T05:32:21.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-16T05:32:21.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.581+0000 7f5ff6e00700 1 Processor -- start 2024-04-16T05:32:21.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.581+0000 7f5ff6e00700 1 -- start start 2024-04-16T05:32:21.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.581+0000 7f5ff6e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff0103d60 0x7f5ff0104170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:21.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.581+0000 7f5ff6e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff01046b0 con 0x7f5ff0103d60 2024-04-16T05:32:21.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.582+0000 7f5ff5a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff0103d60 0x7f5ff0104170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:21.733 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.582+0000 7f5ff5a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff0103d60 0x7f5ff0104170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42526/0 (socket says 172.21.15.27:42526) 2024-04-16T05:32:21.734 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.582+0000 7f5ff5a00700 1 -- 172.21.15.27:0/1396520189 learned_addr learned my addr 172.21.15.27:0/1396520189 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:21.734 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.582+0000 7f5ff5a00700 1 -- 172.21.15.27:0/1396520189 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff0102380 con 0x7f5ff0103d60 2024-04-16T05:32:21.734 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.583+0000 7f5ff5a00700 1 --2- 172.21.15.27:0/1396520189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff0103d60 0x7f5ff0104170 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5fe40098e0 tx=0x7f5fe4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=364910a814a5838b server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:21.734 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.583+0000 7f5fefe00700 1 -- 172.21.15.27:0/1396520189 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fe400b860 con 0x7f5ff0103d60 2024-04-16T05:32:21.734 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.583+0000 7f5fefe00700 1 -- 172.21.15.27:0/1396520189 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5fe400be90 con 0x7f5ff0103d60 2024-04-16T05:32:21.734 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.583+0000 7f5fefe00700 1 -- 172.21.15.27:0/1396520189 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fe40103c0 con 0x7f5ff0103d60 2024-04-16T05:32:21.734 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.584+0000 7f5ff6e00700 1 -- 172.21.15.27:0/1396520189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff0103d60 msgr2=0x7f5ff0104170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:21.734 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.584+0000 7f5ff6e00700 1 --2- 172.21.15.27:0/1396520189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff0103d60 0x7f5ff0104170 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5fe40098e0 tx=0x7f5fe4009250 comp rx=0 tx=0).stop 2024-04-16T05:32:21.734 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.584+0000 7f5ff6e00700 1 -- 172.21.15.27:0/1396520189 shutdown_connections 2024-04-16T05:32:21.734 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.584+0000 7f5ff6e00700 1 --2- 172.21.15.27:0/1396520189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff0103d60 0x7f5ff0104170 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:21.734 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.584+0000 7f5ff6e00700 1 -- 172.21.15.27:0/1396520189 >> 172.21.15.27:0/1396520189 conn(0x7f5ff00fde90 msgr2=0x7f5ff01002a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:21.734 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.584+0000 7f5ff6e00700 1 -- 172.21.15.27:0/1396520189 shutdown_connections 2024-04-16T05:32:21.734 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.584+0000 7f5ff6e00700 1 -- 172.21.15.27:0/1396520189 wait complete. 2024-04-16T05:32:21.734 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.585+0000 7f5ff6e00700 1 Processor -- start 2024-04-16T05:32:21.734 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.585+0000 7f5ff6e00700 1 -- start start 2024-04-16T05:32:21.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.586+0000 7f5ff6e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff0103d60 0x7f5ff0077b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:21.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.586+0000 7f5ff6e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff00780a0 con 0x7f5ff0103d60 2024-04-16T05:32:21.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.586+0000 7f5ff5a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff0103d60 0x7f5ff0077b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:21.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.586+0000 7f5ff5a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff0103d60 0x7f5ff0077b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42542/0 (socket says 172.21.15.27:42542) 2024-04-16T05:32:21.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.586+0000 7f5ff5a00700 1 -- 172.21.15.27:0/3924701955 learned_addr learned my addr 172.21.15.27:0/3924701955 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:21.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.586+0000 7f5ff5a00700 1 -- 172.21.15.27:0/3924701955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe40094f0 con 0x7f5ff0103d60 2024-04-16T05:32:21.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.587+0000 7f5ff5a00700 1 --2- 172.21.15.27:0/3924701955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff0103d60 0x7f5ff0077b60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5fe40090b0 tx=0x7f5fe4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:21.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.587+0000 7f5fed600700 1 -- 172.21.15.27:0/3924701955 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fe4010840 con 0x7f5ff0103d60 2024-04-16T05:32:21.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.587+0000 7f5fed600700 1 -- 172.21.15.27:0/3924701955 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5fe4010e70 con 0x7f5ff0103d60 2024-04-16T05:32:21.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.587+0000 7f5fed600700 1 -- 172.21.15.27:0/3924701955 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fe4024e90 con 0x7f5ff0103d60 2024-04-16T05:32:21.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.587+0000 7f5ff6e00700 1 -- 172.21.15.27:0/3924701955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ff0076110 con 0x7f5ff0103d60 2024-04-16T05:32:21.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.587+0000 7f5ff6e00700 1 -- 172.21.15.27:0/3924701955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ff0076600 con 0x7f5ff0103d60 2024-04-16T05:32:21.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.589+0000 7f5ff6e00700 1 -- 172.21.15.27:0/3924701955 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ff005f5f0 con 0x7f5ff0103d60 2024-04-16T05:32:21.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.589+0000 7f5fed600700 1 -- 172.21.15.27:0/3924701955 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) v1 ==== 43389+0+0 (secure 0 0 0) 0x7f5fe4024430 con 0x7f5ff0103d60 2024-04-16T05:32:21.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.589+0000 7f5fed600700 1 --2- 172.21.15.27:0/3924701955 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f5fd80368a0 0x7f5fd8038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:21.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.590+0000 7f5fed600700 1 -- 172.21.15.27:0/3924701955 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5fe404a9d0 con 0x7f5ff0103d60 2024-04-16T05:32:21.735 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.590+0000 7f5ff5000700 1 --2- 172.21.15.27:0/3924701955 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f5fd80368a0 0x7f5fd8038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:21.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.591+0000 7f5ff5000700 1 --2- 172.21.15.27:0/3924701955 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f5fd80368a0 0x7f5fd8038d50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc009990 tx=0x7f5fdc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:21.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.597+0000 7f5fed600700 1 -- 172.21.15.27:0/3924701955 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f5fe402c930 con 0x7f5ff0103d60 2024-04-16T05:32:21.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.819+0000 7f5ff6e00700 1 -- 172.21.15.27:0/3924701955 --> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ff0076b90 con 0x7f5fd80368a0 2024-04-16T05:32:21.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.915+0000 7f5fed600700 1 -- 172.21.15.27:0/3924701955 <== mgr.14162 v2:172.21.15.27:6800/2700787222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f5ff0076b90 con 0x7f5fd80368a0 2024-04-16T05:32:21.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.918+0000 7f5ff6e00700 1 -- 172.21.15.27:0/3924701955 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f5fd80368a0 msgr2=0x7f5fd8038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:21.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.918+0000 7f5ff6e00700 1 --2- 172.21.15.27:0/3924701955 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f5fd80368a0 0x7f5fd8038d50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc009990 tx=0x7f5fdc006e40 comp rx=0 tx=0).stop 2024-04-16T05:32:21.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.918+0000 7f5ff6e00700 1 -- 172.21.15.27:0/3924701955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff0103d60 msgr2=0x7f5ff0077b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:21.736 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.918+0000 7f5ff6e00700 1 --2- 172.21.15.27:0/3924701955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff0103d60 0x7f5ff0077b60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5fe40090b0 tx=0x7f5fe4009250 comp rx=0 tx=0).stop 2024-04-16T05:32:21.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.919+0000 7f5ff6e00700 1 -- 172.21.15.27:0/3924701955 shutdown_connections 2024-04-16T05:32:21.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.919+0000 7f5ff6e00700 1 --2- 172.21.15.27:0/3924701955 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f5fd80368a0 0x7f5fd8038d50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:21.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.919+0000 7f5ff6e00700 1 --2- 172.21.15.27:0/3924701955 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ff0103d60 0x7f5ff0077b60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:21.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.919+0000 7f5ff6e00700 1 -- 172.21.15.27:0/3924701955 >> 172.21.15.27:0/3924701955 conn(0x7f5ff00fde90 msgr2=0x7f5ff00fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:21.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.919+0000 7f5ff6e00700 1 -- 172.21.15.27:0/3924701955 shutdown_connections 2024-04-16T05:32:21.737 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:20.919+0000 7f5ff6e00700 1 -- 172.21.15.27:0/3924701955 wait complete. 2024-04-16T05:32:21.737 INFO:teuthology.orchestra.run.smithi027.stdout:Creating initial admin user... 2024-04-16T05:32:22.053 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:21 smithi027 ceph-mon[29206]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:32:22.053 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:21 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:22.053 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:21 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:22.053 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:21 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:23.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:22 smithi027 ceph-mon[29206]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:32:23.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:22 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:24.419 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Hrcrt5HuA1EFu8/HehdAN./2UwUuR4i.d4uofAL53eex7ZDVLD04O", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713245542, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-16T05:32:24.420 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.345+0000 7fe7b8400700 1 Processor -- start 2024-04-16T05:32:24.420 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.346+0000 7fe7b8400700 1 -- start start 2024-04-16T05:32:24.420 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.346+0000 7fe7b8400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7b0077ba0 0x7fe7b0076150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:24.420 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.346+0000 7fe7b8400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b0077fb0 con 0x7fe7b0077ba0 2024-04-16T05:32:24.420 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.346+0000 7fe7b5a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7b0077ba0 0x7fe7b0076150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:24.423 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.347+0000 7fe7b5a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7b0077ba0 0x7fe7b0076150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42554/0 (socket says 172.21.15.27:42554) 2024-04-16T05:32:24.423 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.347+0000 7fe7b5a00700 1 -- 172.21.15.27:0/1559768159 learned_addr learned my addr 172.21.15.27:0/1559768159 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:24.423 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.347+0000 7fe7b5a00700 1 -- 172.21.15.27:0/1559768159 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b0076690 con 0x7fe7b0077ba0 2024-04-16T05:32:24.423 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.347+0000 7fe7b5a00700 1 --2- 172.21.15.27:0/1559768159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7b0077ba0 0x7fe7b0076150 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe7a40098e0 tx=0x7fe7a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a914c0e67f8035 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:24.423 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.348+0000 7fe7afe00700 1 -- 172.21.15.27:0/1559768159 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7a400b860 con 0x7fe7b0077ba0 2024-04-16T05:32:24.423 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.348+0000 7fe7afe00700 1 -- 172.21.15.27:0/1559768159 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe7a400be90 con 0x7fe7b0077ba0 2024-04-16T05:32:24.423 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.348+0000 7fe7afe00700 1 -- 172.21.15.27:0/1559768159 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7a40103c0 con 0x7fe7b0077ba0 2024-04-16T05:32:24.423 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.348+0000 7fe7b8400700 1 -- 172.21.15.27:0/1559768159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7b0077ba0 msgr2=0x7fe7b0076150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:24.423 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.348+0000 7fe7b8400700 1 --2- 172.21.15.27:0/1559768159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7b0077ba0 0x7fe7b0076150 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe7a40098e0 tx=0x7fe7a4009250 comp rx=0 tx=0).stop 2024-04-16T05:32:24.423 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.349+0000 7fe7b8400700 1 -- 172.21.15.27:0/1559768159 shutdown_connections 2024-04-16T05:32:24.423 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.349+0000 7fe7b8400700 1 --2- 172.21.15.27:0/1559768159 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7b0077ba0 0x7fe7b0076150 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:24.423 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.349+0000 7fe7b8400700 1 -- 172.21.15.27:0/1559768159 >> 172.21.15.27:0/1559768159 conn(0x7fe7b00fe0a0 msgr2=0x7fe7b01004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:24.424 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.349+0000 7fe7b8400700 1 -- 172.21.15.27:0/1559768159 shutdown_connections 2024-04-16T05:32:24.424 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.349+0000 7fe7b8400700 1 -- 172.21.15.27:0/1559768159 wait complete. 2024-04-16T05:32:24.424 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.350+0000 7fe7b8400700 1 Processor -- start 2024-04-16T05:32:24.424 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.350+0000 7fe7b8400700 1 -- start start 2024-04-16T05:32:24.424 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.350+0000 7fe7b8400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7b0077ba0 0x7fe7b0113860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:24.424 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.350+0000 7fe7b8400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b0113da0 con 0x7fe7b0077ba0 2024-04-16T05:32:24.424 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.351+0000 7fe7b5a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7b0077ba0 0x7fe7b0113860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:24.424 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.351+0000 7fe7b5a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7b0077ba0 0x7fe7b0113860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42564/0 (socket says 172.21.15.27:42564) 2024-04-16T05:32:24.424 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.351+0000 7fe7b5a00700 1 -- 172.21.15.27:0/1711795473 learned_addr learned my addr 172.21.15.27:0/1711795473 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:24.424 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.351+0000 7fe7b5a00700 1 -- 172.21.15.27:0/1711795473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a40094f0 con 0x7fe7b0077ba0 2024-04-16T05:32:24.424 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.352+0000 7fe7b5a00700 1 --2- 172.21.15.27:0/1711795473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7b0077ba0 0x7fe7b0113860 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe7a40105b0 tx=0x7fe7a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:24.424 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.352+0000 7fe7ad600700 1 -- 172.21.15.27:0/1711795473 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7a4010840 con 0x7fe7b0077ba0 2024-04-16T05:32:24.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.352+0000 7fe7ad600700 1 -- 172.21.15.27:0/1711795473 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe7a4010e70 con 0x7fe7b0077ba0 2024-04-16T05:32:24.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.352+0000 7fe7ad600700 1 -- 172.21.15.27:0/1711795473 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7a401ae30 con 0x7fe7b0077ba0 2024-04-16T05:32:24.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.352+0000 7fe7b8400700 1 -- 172.21.15.27:0/1711795473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b0113fa0 con 0x7fe7b0077ba0 2024-04-16T05:32:24.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.352+0000 7fe7b8400700 1 -- 172.21.15.27:0/1711795473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b0114b10 con 0x7fe7b0077ba0 2024-04-16T05:32:24.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.353+0000 7fe7ad600700 1 -- 172.21.15.27:0/1711795473 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 12) v1 ==== 43389+0+0 (secure 0 0 0) 0x7fe7a40109a0 con 0x7fe7b0077ba0 2024-04-16T05:32:24.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.353+0000 7fe7b8400700 1 -- 172.21.15.27:0/1711795473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b005f5f0 con 0x7fe7b0077ba0 2024-04-16T05:32:24.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.354+0000 7fe7ad600700 1 --2- 172.21.15.27:0/1711795473 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fe7980368a0 0x7fe798038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:24.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.354+0000 7fe7ad600700 1 -- 172.21.15.27:0/1711795473 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe7a404a3f0 con 0x7fe7b0077ba0 2024-04-16T05:32:24.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.354+0000 7fe7b5000700 1 --2- 172.21.15.27:0/1711795473 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fe7980368a0 0x7fe798038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:24.427 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.355+0000 7fe7b5000700 1 --2- 172.21.15.27:0/1711795473 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fe7980368a0 0x7fe798038d50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe79c009990 tx=0x7fe79c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:24.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.359+0000 7fe7ad600700 1 -- 172.21.15.27:0/1711795473 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fe7a402b4d0 con 0x7fe7b0077ba0 2024-04-16T05:32:24.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.599+0000 7fe7b8400700 1 -- 172.21.15.27:0/1711795473 --> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fe7b005f410 con 0x7fe7980368a0 2024-04-16T05:32:24.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.891+0000 7fe7ad600700 1 -- 172.21.15.27:0/1711795473 <== mgr.14162 v2:172.21.15.27:6800/2700787222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fe7b005f410 con 0x7fe7980368a0 2024-04-16T05:32:24.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.895+0000 7fe7b8400700 1 -- 172.21.15.27:0/1711795473 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fe7980368a0 msgr2=0x7fe798038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:24.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.895+0000 7fe7b8400700 1 --2- 172.21.15.27:0/1711795473 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fe7980368a0 0x7fe798038d50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe79c009990 tx=0x7fe79c006e40 comp rx=0 tx=0).stop 2024-04-16T05:32:24.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.895+0000 7fe7b8400700 1 -- 172.21.15.27:0/1711795473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7b0077ba0 msgr2=0x7fe7b0113860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:24.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.895+0000 7fe7b8400700 1 --2- 172.21.15.27:0/1711795473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7b0077ba0 0x7fe7b0113860 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe7a40105b0 tx=0x7fe7a4009250 comp rx=0 tx=0).stop 2024-04-16T05:32:24.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.896+0000 7fe7b8400700 1 -- 172.21.15.27:0/1711795473 shutdown_connections 2024-04-16T05:32:24.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.896+0000 7fe7b8400700 1 --2- 172.21.15.27:0/1711795473 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fe7980368a0 0x7fe798038d50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:24.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.896+0000 7fe7b8400700 1 --2- 172.21.15.27:0/1711795473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7b0077ba0 0x7fe7b0113860 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:24.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.896+0000 7fe7b8400700 1 -- 172.21.15.27:0/1711795473 >> 172.21.15.27:0/1711795473 conn(0x7fe7b00fe0a0 msgr2=0x7fe7b00feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:24.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.896+0000 7fe7b8400700 1 -- 172.21.15.27:0/1711795473 shutdown_connections 2024-04-16T05:32:24.428 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:22.896+0000 7fe7b8400700 1 -- 172.21.15.27:0/1711795473 wait complete. 2024-04-16T05:32:24.428 INFO:teuthology.orchestra.run.smithi027.stdout:Fetching dashboard port number... 2024-04-16T05:32:24.991 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:24 smithi027 ceph-mon[29206]: mgrmap e13: smithi027.qgzypi(active, since 6s) 2024-04-16T05:32:26.000 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stdout 8443 2024-04-16T05:32:26.000 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.055+0000 7fac99c00700 1 Processor -- start 2024-04-16T05:32:26.000 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.056+0000 7fac99c00700 1 -- start start 2024-04-16T05:32:26.000 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.056+0000 7fac99c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9407ca40 0x7fac9407ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:26.000 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.056+0000 7fac99c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9407d420 con 0x7fac9407ca40 2024-04-16T05:32:26.001 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.056+0000 7fac92a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9407ca40 0x7fac9407ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:26.001 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.057+0000 7fac92a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9407ca40 0x7fac9407ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48384/0 (socket says 172.21.15.27:48384) 2024-04-16T05:32:26.001 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.057+0000 7fac92a00700 1 -- 172.21.15.27:0/1740067037 learned_addr learned my addr 172.21.15.27:0/1740067037 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:26.001 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.057+0000 7fac92a00700 1 -- 172.21.15.27:0/1740067037 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9407d560 con 0x7fac9407ca40 2024-04-16T05:32:26.001 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.057+0000 7fac92a00700 1 --2- 172.21.15.27:0/1740067037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9407ca40 0x7fac9407ce50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fac840098e0 tx=0x7fac84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b04dd85ecc44a29 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:26.001 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.058+0000 7fac91600700 1 -- 172.21.15.27:0/1740067037 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac8400b860 con 0x7fac9407ca40 2024-04-16T05:32:26.001 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.058+0000 7fac91600700 1 -- 172.21.15.27:0/1740067037 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fac8400be90 con 0x7fac9407ca40 2024-04-16T05:32:26.001 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.058+0000 7fac91600700 1 -- 172.21.15.27:0/1740067037 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac840103c0 con 0x7fac9407ca40 2024-04-16T05:32:26.003 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.058+0000 7fac99c00700 1 -- 172.21.15.27:0/1740067037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9407ca40 msgr2=0x7fac9407ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:26.003 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.058+0000 7fac99c00700 1 --2- 172.21.15.27:0/1740067037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9407ca40 0x7fac9407ce50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fac840098e0 tx=0x7fac84009250 comp rx=0 tx=0).stop 2024-04-16T05:32:26.003 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.058+0000 7fac99c00700 1 -- 172.21.15.27:0/1740067037 shutdown_connections 2024-04-16T05:32:26.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.058+0000 7fac99c00700 1 --2- 172.21.15.27:0/1740067037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9407ca40 0x7fac9407ce50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:26.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.058+0000 7fac99c00700 1 -- 172.21.15.27:0/1740067037 >> 172.21.15.27:0/1740067037 conn(0x7fac94078350 msgr2=0x7fac9407a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:26.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.058+0000 7fac99c00700 1 -- 172.21.15.27:0/1740067037 shutdown_connections 2024-04-16T05:32:26.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.059+0000 7fac99c00700 1 -- 172.21.15.27:0/1740067037 wait complete. 2024-04-16T05:32:26.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.059+0000 7fac99c00700 1 Processor -- start 2024-04-16T05:32:26.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.060+0000 7fac99c00700 1 -- start start 2024-04-16T05:32:26.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.060+0000 7fac99c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9407ca40 0x7fac94117c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:26.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.060+0000 7fac99c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac94118150 con 0x7fac9407ca40 2024-04-16T05:32:26.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.060+0000 7fac92a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9407ca40 0x7fac94117c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:26.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.060+0000 7fac92a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9407ca40 0x7fac94117c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48386/0 (socket says 172.21.15.27:48386) 2024-04-16T05:32:26.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.060+0000 7fac92a00700 1 -- 172.21.15.27:0/3483500153 learned_addr learned my addr 172.21.15.27:0/3483500153 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:26.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.061+0000 7fac92a00700 1 -- 172.21.15.27:0/3483500153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac840094f0 con 0x7fac9407ca40 2024-04-16T05:32:26.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.061+0000 7fac92a00700 1 --2- 172.21.15.27:0/3483500153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9407ca40 0x7fac94117c10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fac84006b00 tx=0x7fac84009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:26.004 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.061+0000 7fac8aa00700 1 -- 172.21.15.27:0/3483500153 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac84010840 con 0x7fac9407ca40 2024-04-16T05:32:26.005 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.061+0000 7fac8aa00700 1 -- 172.21.15.27:0/3483500153 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fac84010e70 con 0x7fac9407ca40 2024-04-16T05:32:26.007 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.061+0000 7fac8aa00700 1 -- 172.21.15.27:0/3483500153 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac8401ae70 con 0x7fac9407ca40 2024-04-16T05:32:26.007 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.061+0000 7fac99c00700 1 -- 172.21.15.27:0/3483500153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac94118350 con 0x7fac9407ca40 2024-04-16T05:32:26.007 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.062+0000 7fac99c00700 1 -- 172.21.15.27:0/3483500153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac94118ec0 con 0x7fac9407ca40 2024-04-16T05:32:26.007 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.063+0000 7fac8aa00700 1 -- 172.21.15.27:0/3483500153 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7fac8401a780 con 0x7fac9407ca40 2024-04-16T05:32:26.007 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.063+0000 7fac99c00700 1 -- 172.21.15.27:0/3483500153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac9405f5f0 con 0x7fac9407ca40 2024-04-16T05:32:26.007 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.063+0000 7fac8aa00700 1 --2- 172.21.15.27:0/3483500153 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fac78036990 0x7fac78038e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:26.008 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.063+0000 7fac8aa00700 1 -- 172.21.15.27:0/3483500153 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fac8404ae30 con 0x7fac9407ca40 2024-04-16T05:32:26.008 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.064+0000 7fac92000700 1 --2- 172.21.15.27:0/3483500153 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fac78036990 0x7fac78038e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:26.008 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.064+0000 7fac92000700 1 --2- 172.21.15.27:0/3483500153 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fac78036990 0x7fac78038e40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fac80009990 tx=0x7fac80006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:26.008 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.069+0000 7fac8aa00700 1 -- 172.21.15.27:0/3483500153 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fac8404f070 con 0x7fac9407ca40 2024-04-16T05:32:26.008 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.284+0000 7fac99c00700 1 -- 172.21.15.27:0/3483500153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fac9404bed0 con 0x7fac9407ca40 2024-04-16T05:32:26.008 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.285+0000 7fac8aa00700 1 -- 172.21.15.27:0/3483500153 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fac8401a430 con 0x7fac9407ca40 2024-04-16T05:32:26.008 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.288+0000 7fac99c00700 1 -- 172.21.15.27:0/3483500153 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fac78036990 msgr2=0x7fac78038e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:26.008 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.288+0000 7fac99c00700 1 --2- 172.21.15.27:0/3483500153 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fac78036990 0x7fac78038e40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fac80009990 tx=0x7fac80006e40 comp rx=0 tx=0).stop 2024-04-16T05:32:26.008 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.288+0000 7fac99c00700 1 -- 172.21.15.27:0/3483500153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9407ca40 msgr2=0x7fac94117c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:26.008 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.288+0000 7fac99c00700 1 --2- 172.21.15.27:0/3483500153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9407ca40 0x7fac94117c10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fac84006b00 tx=0x7fac84009250 comp rx=0 tx=0).stop 2024-04-16T05:32:26.008 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.289+0000 7fac99c00700 1 -- 172.21.15.27:0/3483500153 shutdown_connections 2024-04-16T05:32:26.008 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.289+0000 7fac99c00700 1 --2- 172.21.15.27:0/3483500153 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fac78036990 0x7fac78038e40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:26.008 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.289+0000 7fac99c00700 1 --2- 172.21.15.27:0/3483500153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9407ca40 0x7fac94117c10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:26.008 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.289+0000 7fac99c00700 1 -- 172.21.15.27:0/3483500153 >> 172.21.15.27:0/3483500153 conn(0x7fac94078350 msgr2=0x7fac94079da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:26.008 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.289+0000 7fac99c00700 1 -- 172.21.15.27:0/3483500153 shutdown_connections 2024-04-16T05:32:26.008 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:25.289+0000 7fac99c00700 1 -- 172.21.15.27:0/3483500153 wait complete. 2024-04-16T05:32:26.040 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-16T05:32:26.040 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout disabled 2024-04-16T05:32:26.064 INFO:teuthology.orchestra.run.smithi027.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-16T05:32:26.065 INFO:teuthology.orchestra.run.smithi027.stdout:systemctl: stdout inactive 2024-04-16T05:32:26.065 INFO:teuthology.orchestra.run.smithi027.stdout:firewalld.service is not enabled 2024-04-16T05:32:26.065 INFO:teuthology.orchestra.run.smithi027.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-16T05:32:26.073 INFO:teuthology.orchestra.run.smithi027.stdout:Ceph Dashboard is now available at: 2024-04-16T05:32:26.073 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:32:26.073 INFO:teuthology.orchestra.run.smithi027.stdout: URL: https://smithi027.front.sepia.ceph.com:8443/ 2024-04-16T05:32:26.073 INFO:teuthology.orchestra.run.smithi027.stdout: User: admin 2024-04-16T05:32:26.073 INFO:teuthology.orchestra.run.smithi027.stdout: Password: ra70xl268y 2024-04-16T05:32:26.073 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:32:26.074 INFO:teuthology.orchestra.run.smithi027.stdout:Saving cluster configuration to /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config directory 2024-04-16T05:32:26.076 INFO:teuthology.orchestra.run.smithi027.stdout:Enabling autotune for osd_memory_target 2024-04-16T05:32:26.237 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:25 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/3483500153' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-16T05:32:27.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.710+0000 7f5a36200700 1 Processor -- start 2024-04-16T05:32:27.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.710+0000 7f5a36200700 1 -- start start 2024-04-16T05:32:27.630 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.710+0000 7f5a36200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a30104c20 0x7f5a30105030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:27.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.711+0000 7f5a36200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a30105600 con 0x7f5a30104c20 2024-04-16T05:32:27.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.711+0000 7f5a2f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a30104c20 0x7f5a30105030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:27.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.711+0000 7f5a2f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a30104c20 0x7f5a30105030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48398/0 (socket says 172.21.15.27:48398) 2024-04-16T05:32:27.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.711+0000 7f5a2f400700 1 -- 172.21.15.27:0/4075803717 learned_addr learned my addr 172.21.15.27:0/4075803717 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:27.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.712+0000 7f5a2f400700 1 -- 172.21.15.27:0/4075803717 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a30105740 con 0x7f5a30104c20 2024-04-16T05:32:27.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.712+0000 7f5a2f400700 1 --2- 172.21.15.27:0/4075803717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a30104c20 0x7f5a30105030 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5a200098e0 tx=0x7f5a20009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd65398f8bbb29af server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:27.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.712+0000 7f5a2e000700 1 -- 172.21.15.27:0/4075803717 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a2000b860 con 0x7f5a30104c20 2024-04-16T05:32:27.636 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.712+0000 7f5a2e000700 1 -- 172.21.15.27:0/4075803717 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a2000be90 con 0x7f5a30104c20 2024-04-16T05:32:27.637 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.713+0000 7f5a2e000700 1 -- 172.21.15.27:0/4075803717 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a200103c0 con 0x7f5a30104c20 2024-04-16T05:32:27.637 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.713+0000 7f5a36200700 1 -- 172.21.15.27:0/4075803717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a30104c20 msgr2=0x7f5a30105030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:27.637 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.713+0000 7f5a36200700 1 --2- 172.21.15.27:0/4075803717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a30104c20 0x7f5a30105030 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5a200098e0 tx=0x7f5a20009250 comp rx=0 tx=0).stop 2024-04-16T05:32:27.637 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.713+0000 7f5a36200700 1 -- 172.21.15.27:0/4075803717 shutdown_connections 2024-04-16T05:32:27.637 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.713+0000 7f5a36200700 1 --2- 172.21.15.27:0/4075803717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a30104c20 0x7f5a30105030 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:27.637 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.713+0000 7f5a36200700 1 -- 172.21.15.27:0/4075803717 >> 172.21.15.27:0/4075803717 conn(0x7f5a30078350 msgr2=0x7f5a30078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:27.637 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.713+0000 7f5a36200700 1 -- 172.21.15.27:0/4075803717 shutdown_connections 2024-04-16T05:32:27.637 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.714+0000 7f5a36200700 1 -- 172.21.15.27:0/4075803717 wait complete. 2024-04-16T05:32:27.637 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.714+0000 7f5a36200700 1 Processor -- start 2024-04-16T05:32:27.637 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.715+0000 7f5a36200700 1 -- start start 2024-04-16T05:32:27.637 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.715+0000 7f5a36200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a30104c20 0x7f5a30113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:27.637 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.715+0000 7f5a36200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a30113d60 con 0x7f5a30104c20 2024-04-16T05:32:27.637 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.715+0000 7f5a2f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a30104c20 0x7f5a30113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:27.638 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.715+0000 7f5a2f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a30104c20 0x7f5a30113820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48400/0 (socket says 172.21.15.27:48400) 2024-04-16T05:32:27.638 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.715+0000 7f5a2f400700 1 -- 172.21.15.27:0/3217492666 learned_addr learned my addr 172.21.15.27:0/3217492666 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:27.638 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.716+0000 7f5a2f400700 1 -- 172.21.15.27:0/3217492666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a200094f0 con 0x7f5a30104c20 2024-04-16T05:32:27.638 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.716+0000 7f5a2f400700 1 --2- 172.21.15.27:0/3217492666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a30104c20 0x7f5a30113820 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5a20006b00 tx=0x7f5a20009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:27.638 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.716+0000 7f5a27400700 1 -- 172.21.15.27:0/3217492666 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a20010840 con 0x7f5a30104c20 2024-04-16T05:32:27.638 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.716+0000 7f5a27400700 1 -- 172.21.15.27:0/3217492666 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a20010e70 con 0x7f5a30104c20 2024-04-16T05:32:27.638 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.716+0000 7f5a36200700 1 -- 172.21.15.27:0/3217492666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a30113f60 con 0x7f5a30104c20 2024-04-16T05:32:27.638 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.716+0000 7f5a27400700 1 -- 172.21.15.27:0/3217492666 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a200243f0 con 0x7f5a30104c20 2024-04-16T05:32:27.638 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.716+0000 7f5a36200700 1 -- 172.21.15.27:0/3217492666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a30114ad0 con 0x7f5a30104c20 2024-04-16T05:32:27.638 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.718+0000 7f5a27400700 1 -- 172.21.15.27:0/3217492666 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f5a200109a0 con 0x7f5a30104c20 2024-04-16T05:32:27.638 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.718+0000 7f5a36200700 1 -- 172.21.15.27:0/3217492666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a3005f5f0 con 0x7f5a30104c20 2024-04-16T05:32:27.638 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.718+0000 7f5a27400700 1 --2- 172.21.15.27:0/3217492666 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f5a14036990 0x7f5a14038e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:27.638 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.718+0000 7f5a27400700 1 -- 172.21.15.27:0/3217492666 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5a2004aa50 con 0x7f5a30104c20 2024-04-16T05:32:27.638 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.719+0000 7f5a2ea00700 1 --2- 172.21.15.27:0/3217492666 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f5a14036990 0x7f5a14038e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:27.638 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.720+0000 7f5a2ea00700 1 --2- 172.21.15.27:0/3217492666 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f5a14036990 0x7f5a14038e40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c009990 tx=0x7f5a1c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:27.639 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.724+0000 7f5a27400700 1 -- 172.21.15.27:0/3217492666 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f5a20018b40 con 0x7f5a30104c20 2024-04-16T05:32:27.639 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.942+0000 7f5a36200700 1 -- 172.21.15.27:0/3217492666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f5a3004bed0 con 0x7f5a30104c20 2024-04-16T05:32:27.639 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.943+0000 7f5a27400700 1 -- 172.21.15.27:0/3217492666 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f5a20018b40 con 0x7f5a30104c20 2024-04-16T05:32:27.639 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.946+0000 7f5a36200700 1 -- 172.21.15.27:0/3217492666 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f5a14036990 msgr2=0x7f5a14038e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:27.639 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.946+0000 7f5a36200700 1 --2- 172.21.15.27:0/3217492666 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f5a14036990 0x7f5a14038e40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c009990 tx=0x7f5a1c006e40 comp rx=0 tx=0).stop 2024-04-16T05:32:27.639 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.946+0000 7f5a36200700 1 -- 172.21.15.27:0/3217492666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a30104c20 msgr2=0x7f5a30113820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:27.639 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.946+0000 7f5a36200700 1 --2- 172.21.15.27:0/3217492666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a30104c20 0x7f5a30113820 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5a20006b00 tx=0x7f5a20009250 comp rx=0 tx=0).stop 2024-04-16T05:32:27.639 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.946+0000 7f5a36200700 1 -- 172.21.15.27:0/3217492666 shutdown_connections 2024-04-16T05:32:27.639 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.946+0000 7f5a36200700 1 --2- 172.21.15.27:0/3217492666 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f5a14036990 0x7f5a14038e40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:27.639 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.946+0000 7f5a36200700 1 --2- 172.21.15.27:0/3217492666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a30104c20 0x7f5a30113820 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:27.639 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.946+0000 7f5a36200700 1 -- 172.21.15.27:0/3217492666 >> 172.21.15.27:0/3217492666 conn(0x7f5a30078350 msgr2=0x7f5a30102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:27.639 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.947+0000 7f5a36200700 1 -- 172.21.15.27:0/3217492666 shutdown_connections 2024-04-16T05:32:27.639 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:26.947+0000 7f5a36200700 1 -- 172.21.15.27:0/3217492666 wait complete. 2024-04-16T05:32:29.249 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.258+0000 7f09a9000700 1 Processor -- start 2024-04-16T05:32:29.249 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.259+0000 7f09a9000700 1 -- start start 2024-04-16T05:32:29.249 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.259+0000 7f09a9000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09a40712c0 0x7f09a4107920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:29.249 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.259+0000 7f09a9000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a4103310 con 0x7f09a40712c0 2024-04-16T05:32:29.249 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.260+0000 7f09a1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09a40712c0 0x7f09a4107920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:29.249 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.260+0000 7f09a1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09a40712c0 0x7f09a4107920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48406/0 (socket says 172.21.15.27:48406) 2024-04-16T05:32:29.249 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.260+0000 7f09a1e00700 1 -- 172.21.15.27:0/867564962 learned_addr learned my addr 172.21.15.27:0/867564962 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:29.249 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.260+0000 7f09a1e00700 1 -- 172.21.15.27:0/867564962 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a4107e60 con 0x7f09a40712c0 2024-04-16T05:32:29.250 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.260+0000 7f09a1e00700 1 --2- 172.21.15.27:0/867564962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09a40712c0 0x7f09a4107920 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f09940098e0 tx=0x7f0994009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9934a8ce3b9cdfe3 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:29.250 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.261+0000 7f09a0a00700 1 -- 172.21.15.27:0/867564962 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f099400b860 con 0x7f09a40712c0 2024-04-16T05:32:29.250 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.261+0000 7f09a0a00700 1 -- 172.21.15.27:0/867564962 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f099400be90 con 0x7f09a40712c0 2024-04-16T05:32:29.250 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.261+0000 7f09a0a00700 1 -- 172.21.15.27:0/867564962 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09940103c0 con 0x7f09a40712c0 2024-04-16T05:32:29.250 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.261+0000 7f09a9000700 1 -- 172.21.15.27:0/867564962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09a40712c0 msgr2=0x7f09a4107920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:29.250 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.261+0000 7f09a9000700 1 --2- 172.21.15.27:0/867564962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09a40712c0 0x7f09a4107920 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f09940098e0 tx=0x7f0994009250 comp rx=0 tx=0).stop 2024-04-16T05:32:29.250 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.262+0000 7f09a9000700 1 -- 172.21.15.27:0/867564962 shutdown_connections 2024-04-16T05:32:29.250 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.262+0000 7f09a9000700 1 --2- 172.21.15.27:0/867564962 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09a40712c0 0x7f09a4107920 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:29.250 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.262+0000 7f09a9000700 1 -- 172.21.15.27:0/867564962 >> 172.21.15.27:0/867564962 conn(0x7f09a40fcfd0 msgr2=0x7f09a40ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:29.250 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.262+0000 7f09a9000700 1 -- 172.21.15.27:0/867564962 shutdown_connections 2024-04-16T05:32:29.250 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.262+0000 7f09a9000700 1 -- 172.21.15.27:0/867564962 wait complete. 2024-04-16T05:32:29.250 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.263+0000 7f09a9000700 1 Processor -- start 2024-04-16T05:32:29.250 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.263+0000 7f09a9000700 1 -- start start 2024-04-16T05:32:29.251 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.263+0000 7f09a9000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09a40712c0 0x7f09a4113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:29.251 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.263+0000 7f09a9000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a4113db0 con 0x7f09a40712c0 2024-04-16T05:32:29.251 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.263+0000 7f09a1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09a40712c0 0x7f09a4113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:29.251 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.264+0000 7f09a1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09a40712c0 0x7f09a4113870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48408/0 (socket says 172.21.15.27:48408) 2024-04-16T05:32:29.251 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.264+0000 7f09a1e00700 1 -- 172.21.15.27:0/2321598489 learned_addr learned my addr 172.21.15.27:0/2321598489 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:29.251 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.264+0000 7f09a1e00700 1 -- 172.21.15.27:0/2321598489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09940094f0 con 0x7f09a40712c0 2024-04-16T05:32:29.251 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.264+0000 7f09a1e00700 1 --2- 172.21.15.27:0/2321598489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09a40712c0 0x7f09a4113870 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f09940105b0 tx=0x7f0994009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:29.251 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.264+0000 7f099a000700 1 -- 172.21.15.27:0/2321598489 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0994010840 con 0x7f09a40712c0 2024-04-16T05:32:29.251 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.265+0000 7f099a000700 1 -- 172.21.15.27:0/2321598489 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0994010e70 con 0x7f09a40712c0 2024-04-16T05:32:29.251 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.265+0000 7f09a9000700 1 -- 172.21.15.27:0/2321598489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a4113fb0 con 0x7f09a40712c0 2024-04-16T05:32:29.251 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.265+0000 7f099a000700 1 -- 172.21.15.27:0/2321598489 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f099401ae30 con 0x7f09a40712c0 2024-04-16T05:32:29.251 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.265+0000 7f09a9000700 1 -- 172.21.15.27:0/2321598489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a4114b20 con 0x7f09a40712c0 2024-04-16T05:32:29.252 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.266+0000 7f099a000700 1 -- 172.21.15.27:0/2321598489 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f099401a430 con 0x7f09a40712c0 2024-04-16T05:32:29.252 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.266+0000 7f09a9000700 1 -- 172.21.15.27:0/2321598489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a405f5f0 con 0x7f09a40712c0 2024-04-16T05:32:29.252 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.266+0000 7f099a000700 1 --2- 172.21.15.27:0/2321598489 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f0988036940 0x7f0988038df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:29.252 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.266+0000 7f099a000700 1 -- 172.21.15.27:0/2321598489 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f099404a380 con 0x7f09a40712c0 2024-04-16T05:32:29.252 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.267+0000 7f09a1400700 1 --2- 172.21.15.27:0/2321598489 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f0988036940 0x7f0988038df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:29.252 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.267+0000 7f09a1400700 1 --2- 172.21.15.27:0/2321598489 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f0988036940 0x7f0988038df0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0990009990 tx=0x7f0990006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:29.252 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.271+0000 7f099a000700 1 -- 172.21.15.27:0/2321598489 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f099402b8c0 con 0x7f09a40712c0 2024-04-16T05:32:29.252 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.600+0000 7f09a9000700 1 -- 172.21.15.27:0/2321598489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f09a404bed0 con 0x7f09a40712c0 2024-04-16T05:32:29.252 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.603+0000 7f099a000700 1 -- 172.21.15.27:0/2321598489 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f099402b2f0 con 0x7f09a40712c0 2024-04-16T05:32:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.606+0000 7f09a9000700 1 -- 172.21.15.27:0/2321598489 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f0988036940 msgr2=0x7f0988038df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.606+0000 7f09a9000700 1 --2- 172.21.15.27:0/2321598489 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f0988036940 0x7f0988038df0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0990009990 tx=0x7f0990006e40 comp rx=0 tx=0).stop 2024-04-16T05:32:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.606+0000 7f09a9000700 1 -- 172.21.15.27:0/2321598489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09a40712c0 msgr2=0x7f09a4113870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.606+0000 7f09a9000700 1 --2- 172.21.15.27:0/2321598489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09a40712c0 0x7f09a4113870 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f09940105b0 tx=0x7f0994009250 comp rx=0 tx=0).stop 2024-04-16T05:32:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.606+0000 7f09a9000700 1 -- 172.21.15.27:0/2321598489 shutdown_connections 2024-04-16T05:32:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.606+0000 7f09a9000700 1 --2- 172.21.15.27:0/2321598489 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f0988036940 0x7f0988038df0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.606+0000 7f09a9000700 1 --2- 172.21.15.27:0/2321598489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09a40712c0 0x7f09a4113870 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.606+0000 7f09a9000700 1 -- 172.21.15.27:0/2321598489 >> 172.21.15.27:0/2321598489 conn(0x7f09a40fcfd0 msgr2=0x7f09a40fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.606+0000 7f09a9000700 1 -- 172.21.15.27:0/2321598489 shutdown_connections 2024-04-16T05:32:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr 2024-04-16T05:32:28.607+0000 7f09a9000700 1 -- 172.21.15.27:0/2321598489 wait complete. 2024-04-16T05:32:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-16T05:32:29.253 INFO:teuthology.orchestra.run.smithi027.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-16T05:32:29.253 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:32:29.254 INFO:teuthology.orchestra.run.smithi027.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-16T05:32:29.254 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:32:29.254 INFO:teuthology.orchestra.run.smithi027.stdout:Or, if you are only running a single cluster on this host: 2024-04-16T05:32:29.254 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:32:29.254 INFO:teuthology.orchestra.run.smithi027.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-16T05:32:29.254 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:32:29.254 INFO:teuthology.orchestra.run.smithi027.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-16T05:32:29.254 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:32:29.254 INFO:teuthology.orchestra.run.smithi027.stdout: ceph telemetry on 2024-04-16T05:32:29.254 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:32:29.254 INFO:teuthology.orchestra.run.smithi027.stdout:For more information see: 2024-04-16T05:32:29.254 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:32:29.254 INFO:teuthology.orchestra.run.smithi027.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-16T05:32:29.255 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:32:29.255 INFO:teuthology.orchestra.run.smithi027.stdout:Bootstrap complete. 2024-04-16T05:32:29.354 INFO:tasks.cephadm:Fetching config... 2024-04-16T05:32:29.354 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-16T05:32:29.354 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-16T05:32:29.410 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-16T05:32:29.410 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-16T05:32:29.410 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-16T05:32:29.451 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-16T05:32:29.452 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-16T05:32:29.452 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/keyring of=/dev/stdout 2024-04-16T05:32:29.576 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-16T05:32:29.576 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-16T05:32:29.577 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-16T05:32:29.671 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-16T05:32:29.671 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIwdQH4aotIZd9sP60hGpybsp0U0UEe/hy0wcbh+eT0SDhe2RhSv2RbpvBReQDxqlBo72AYaVPV2fpVezYRLbid0tRka6jbYtDITICfEi3dsM0hx+wns2fFahNEAGgcOYnO5oWl4J2GAzXtvU8NVXCYE/H4MRWjfSi3I+j/hV/CVOb6HeQ/q4skiJ0CfMmo+KUd7dgs3KAparIXq2HEzr+EOaUCXG34rzn7Yl7K8UhKCbBOBZw2ut/KTlK3+7rjh4q/3vTpVC1kr8AjeGHQIRovJ+TD2MOQXKO2T2Eo3qE5vTeAqhIbL67n7wzGk7w4EnbQ6zWJ1rvBPZ7hUvv34AAi4Wj0swFO0xCoJfh7izLrew5OS43OwvQCxB+GjprIGpSxlEqM61QDxJOFe2nLrCFWSurZrhb4WwsmqcXIc8UG2HokTlo1tJVMu7qz0iAs1t5WQ7JWLBlqYnqcYWQwux9wyv0/6r/FJVdRtW5ps+idKxNrOXhcbGsT/h5fvRCqYk= ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-16T05:32:29.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:29 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/2321598489' entity='client.admin' 2024-04-16T05:32:29.829 INFO:teuthology.orchestra.run.smithi027.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIwdQH4aotIZd9sP60hGpybsp0U0UEe/hy0wcbh+eT0SDhe2RhSv2RbpvBReQDxqlBo72AYaVPV2fpVezYRLbid0tRka6jbYtDITICfEi3dsM0hx+wns2fFahNEAGgcOYnO5oWl4J2GAzXtvU8NVXCYE/H4MRWjfSi3I+j/hV/CVOb6HeQ/q4skiJ0CfMmo+KUd7dgs3KAparIXq2HEzr+EOaUCXG34rzn7Yl7K8UhKCbBOBZw2ut/KTlK3+7rjh4q/3vTpVC1kr8AjeGHQIRovJ+TD2MOQXKO2T2Eo3qE5vTeAqhIbL67n7wzGk7w4EnbQ6zWJ1rvBPZ7hUvv34AAi4Wj0swFO0xCoJfh7izLrew5OS43OwvQCxB+GjprIGpSxlEqM61QDxJOFe2nLrCFWSurZrhb4WwsmqcXIc8UG2HokTlo1tJVMu7qz0iAs1t5WQ7JWLBlqYnqcYWQwux9wyv0/6r/FJVdRtW5ps+idKxNrOXhcbGsT/h5fvRCqYk= ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:32:29.899 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIwdQH4aotIZd9sP60hGpybsp0U0UEe/hy0wcbh+eT0SDhe2RhSv2RbpvBReQDxqlBo72AYaVPV2fpVezYRLbid0tRka6jbYtDITICfEi3dsM0hx+wns2fFahNEAGgcOYnO5oWl4J2GAzXtvU8NVXCYE/H4MRWjfSi3I+j/hV/CVOb6HeQ/q4skiJ0CfMmo+KUd7dgs3KAparIXq2HEzr+EOaUCXG34rzn7Yl7K8UhKCbBOBZw2ut/KTlK3+7rjh4q/3vTpVC1kr8AjeGHQIRovJ+TD2MOQXKO2T2Eo3qE5vTeAqhIbL67n7wzGk7w4EnbQ6zWJ1rvBPZ7hUvv34AAi4Wj0swFO0xCoJfh7izLrew5OS43OwvQCxB+GjprIGpSxlEqM61QDxJOFe2nLrCFWSurZrhb4WwsmqcXIc8UG2HokTlo1tJVMu7qz0iAs1t5WQ7JWLBlqYnqcYWQwux9wyv0/6r/FJVdRtW5ps+idKxNrOXhcbGsT/h5fvRCqYk= ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-16T05:32:30.025 INFO:teuthology.orchestra.run.smithi122.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIwdQH4aotIZd9sP60hGpybsp0U0UEe/hy0wcbh+eT0SDhe2RhSv2RbpvBReQDxqlBo72AYaVPV2fpVezYRLbid0tRka6jbYtDITICfEi3dsM0hx+wns2fFahNEAGgcOYnO5oWl4J2GAzXtvU8NVXCYE/H4MRWjfSi3I+j/hV/CVOb6HeQ/q4skiJ0CfMmo+KUd7dgs3KAparIXq2HEzr+EOaUCXG34rzn7Yl7K8UhKCbBOBZw2ut/KTlK3+7rjh4q/3vTpVC1kr8AjeGHQIRovJ+TD2MOQXKO2T2Eo3qE5vTeAqhIbL67n7wzGk7w4EnbQ6zWJ1rvBPZ7hUvv34AAi4Wj0swFO0xCoJfh7izLrew5OS43OwvQCxB+GjprIGpSxlEqM61QDxJOFe2nLrCFWSurZrhb4WwsmqcXIc8UG2HokTlo1tJVMu7qz0iAs1t5WQ7JWLBlqYnqcYWQwux9wyv0/6r/FJVdRtW5ps+idKxNrOXhcbGsT/h5fvRCqYk= ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:32:30.089 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-16T05:32:30.573 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:32:33.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.574+0000 7f3130000700 1 -- 172.21.15.27:0/1395054305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3128065d60 msgr2=0x7f31280ff660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:33.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.574+0000 7f3130000700 1 --2- 172.21.15.27:0/1395054305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3128065d60 0x7f31280ff660 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f311c0048c0 tx=0x7f311c009e30 comp rx=0 tx=0).stop 2024-04-16T05:32:33.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.574+0000 7f3130000700 1 -- 172.21.15.27:0/1395054305 shutdown_connections 2024-04-16T05:32:33.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.574+0000 7f3130000700 1 --2- 172.21.15.27:0/1395054305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3128065d60 0x7f31280ff660 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:33.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.574+0000 7f3130000700 1 -- 172.21.15.27:0/1395054305 >> 172.21.15.27:0/1395054305 conn(0x7f31280f70d0 msgr2=0x7f31280f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:33.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.574+0000 7f3130000700 1 -- 172.21.15.27:0/1395054305 shutdown_connections 2024-04-16T05:32:33.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.574+0000 7f3130000700 1 -- 172.21.15.27:0/1395054305 wait complete. 2024-04-16T05:32:33.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.575+0000 7f3130000700 1 Processor -- start 2024-04-16T05:32:33.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.575+0000 7f3130000700 1 -- start start 2024-04-16T05:32:33.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.576+0000 7f3130000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3128065d60 0x7f31280fcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:33.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.576+0000 7f3130000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31280fd4f0 con 0x7f3128065d60 2024-04-16T05:32:33.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.576+0000 7f312d600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3128065d60 0x7f31280fcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:33.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.576+0000 7f312d600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3128065d60 0x7f31280fcfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48432/0 (socket says 172.21.15.27:48432) 2024-04-16T05:32:33.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.576+0000 7f312d600700 1 -- 172.21.15.27:0/2253288583 learned_addr learned my addr 172.21.15.27:0/2253288583 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:33.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.577+0000 7f312d600700 1 -- 172.21.15.27:0/2253288583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311c005040 con 0x7f3128065d60 2024-04-16T05:32:33.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.577+0000 7f312d600700 1 --2- 172.21.15.27:0/2253288583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3128065d60 0x7f31280fcfb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f311c009680 tx=0x7f311c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:33.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.577+0000 7f3125600700 1 -- 172.21.15.27:0/2253288583 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f311c00bed0 con 0x7f3128065d60 2024-04-16T05:32:33.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.577+0000 7f3130000700 1 -- 172.21.15.27:0/2253288583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31280fb5c0 con 0x7f3128065d60 2024-04-16T05:32:33.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.577+0000 7f3125600700 1 -- 172.21.15.27:0/2253288583 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f311c00b5c0 con 0x7f3128065d60 2024-04-16T05:32:33.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.577+0000 7f3125600700 1 -- 172.21.15.27:0/2253288583 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f311c01c5b0 con 0x7f3128065d60 2024-04-16T05:32:33.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.577+0000 7f3130000700 1 -- 172.21.15.27:0/2253288583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31280fbab0 con 0x7f3128065d60 2024-04-16T05:32:33.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.578+0000 7f3130000700 1 -- 172.21.15.27:0/2253288583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f312805f5f0 con 0x7f3128065d60 2024-04-16T05:32:33.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.579+0000 7f3125600700 1 -- 172.21.15.27:0/2253288583 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f311c013420 con 0x7f3128065d60 2024-04-16T05:32:33.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.579+0000 7f3125600700 1 --2- 172.21.15.27:0/2253288583 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f31100368d0 0x7f3110038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:33.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.579+0000 7f3125600700 1 -- 172.21.15.27:0/2253288583 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f311c049930 con 0x7f3128065d60 2024-04-16T05:32:33.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.579+0000 7f312cc00700 1 --2- 172.21.15.27:0/2253288583 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f31100368d0 0x7f3110038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:33.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.580+0000 7f312cc00700 1 --2- 172.21.15.27:0/2253288583 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f31100368d0 0x7f3110038d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3114009990 tx=0x7f3114006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:33.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.585+0000 7f3125600700 1 -- 172.21.15.27:0/2253288583 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f311c013a80 con 0x7f3128065d60 2024-04-16T05:32:33.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.802+0000 7f3130000700 1 -- 172.21.15.27:0/2253288583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f312804bed0 con 0x7f3128065d60 2024-04-16T05:32:33.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.810+0000 7f3125600700 1 -- 172.21.15.27:0/2253288583 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f311c011370 con 0x7f3128065d60 2024-04-16T05:32:33.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.813+0000 7f3130000700 1 -- 172.21.15.27:0/2253288583 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f31100368d0 msgr2=0x7f3110038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:33.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.813+0000 7f3130000700 1 --2- 172.21.15.27:0/2253288583 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f31100368d0 0x7f3110038d80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3114009990 tx=0x7f3114006e40 comp rx=0 tx=0).stop 2024-04-16T05:32:33.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.813+0000 7f3130000700 1 -- 172.21.15.27:0/2253288583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3128065d60 msgr2=0x7f31280fcfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:33.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.813+0000 7f3130000700 1 --2- 172.21.15.27:0/2253288583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3128065d60 0x7f31280fcfb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f311c009680 tx=0x7f311c009e30 comp rx=0 tx=0).stop 2024-04-16T05:32:33.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.813+0000 7f3130000700 1 -- 172.21.15.27:0/2253288583 shutdown_connections 2024-04-16T05:32:33.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.813+0000 7f3130000700 1 --2- 172.21.15.27:0/2253288583 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f31100368d0 0x7f3110038d80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:33.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.813+0000 7f3130000700 1 --2- 172.21.15.27:0/2253288583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3128065d60 0x7f31280fcfb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:33.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.813+0000 7f3130000700 1 -- 172.21.15.27:0/2253288583 >> 172.21.15.27:0/2253288583 conn(0x7f31280f70d0 msgr2=0x7f3128103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:33.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.813+0000 7f3130000700 1 -- 172.21.15.27:0/2253288583 shutdown_connections 2024-04-16T05:32:33.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:33.813+0000 7f3130000700 1 -- 172.21.15.27:0/2253288583 wait complete. 2024-04-16T05:32:34.275 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:33 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:34.275 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:33 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:34.275 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:33 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd/host:smithi027", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:32:34.275 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:33 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:34.275 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:33 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi027", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T05:32:34.275 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:33 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi027", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-16T05:32:34.275 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:33 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:32:34.275 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:33 smithi027 ceph-mon[29206]: Deploying daemon crash.smithi027 on smithi027 2024-04-16T05:32:34.275 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:33 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/2253288583' entity='client.admin' 2024-04-16T05:32:34.688 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-16T05:32:34.688 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-16T05:32:35.173 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:32:38.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.396+0000 7fe157a00700 1 -- 172.21.15.27:0/3640387102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1500fca50 msgr2=0x7fe1500fce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:38.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.396+0000 7fe157a00700 1 --2- 172.21.15.27:0/3640387102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1500fca50 0x7fe1500fce60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe1440048c0 tx=0x7fe144009e30 comp rx=0 tx=0).stop 2024-04-16T05:32:38.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.396+0000 7fe157a00700 1 -- 172.21.15.27:0/3640387102 shutdown_connections 2024-04-16T05:32:38.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.396+0000 7fe157a00700 1 --2- 172.21.15.27:0/3640387102 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1500fca50 0x7fe1500fce60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:38.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.396+0000 7fe157a00700 1 -- 172.21.15.27:0/3640387102 >> 172.21.15.27:0/3640387102 conn(0x7fe1500f81d0 msgr2=0x7fe1500fa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:38.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.396+0000 7fe157a00700 1 -- 172.21.15.27:0/3640387102 shutdown_connections 2024-04-16T05:32:38.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.396+0000 7fe157a00700 1 -- 172.21.15.27:0/3640387102 wait complete. 2024-04-16T05:32:38.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.397+0000 7fe157a00700 1 Processor -- start 2024-04-16T05:32:38.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.397+0000 7fe157a00700 1 -- start start 2024-04-16T05:32:38.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.397+0000 7fe157a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1500fca50 0x7fe15010f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:38.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.397+0000 7fe157a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15010f800 con 0x7fe1500fca50 2024-04-16T05:32:38.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.398+0000 7fe155000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1500fca50 0x7fe15010f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:38.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.398+0000 7fe155000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1500fca50 0x7fe15010f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44380/0 (socket says 172.21.15.27:44380) 2024-04-16T05:32:38.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.398+0000 7fe155000700 1 -- 172.21.15.27:0/2215490453 learned_addr learned my addr 172.21.15.27:0/2215490453 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:38.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.398+0000 7fe155000700 1 -- 172.21.15.27:0/2215490453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe144005040 con 0x7fe1500fca50 2024-04-16T05:32:38.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.398+0000 7fe155000700 1 --2- 172.21.15.27:0/2215490453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1500fca50 0x7fe15010f2c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe1440096b0 tx=0x7fe144009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:38.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.398+0000 7fe14cc00700 1 -- 172.21.15.27:0/2215490453 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe14400bed0 con 0x7fe1500fca50 2024-04-16T05:32:38.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.399+0000 7fe14cc00700 1 -- 172.21.15.27:0/2215490453 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe14400b5c0 con 0x7fe1500fca50 2024-04-16T05:32:38.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.399+0000 7fe14cc00700 1 -- 172.21.15.27:0/2215490453 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe14401c5b0 con 0x7fe1500fca50 2024-04-16T05:32:38.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.399+0000 7fe157a00700 1 -- 172.21.15.27:0/2215490453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15010fa00 con 0x7fe1500fca50 2024-04-16T05:32:38.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.399+0000 7fe157a00700 1 -- 172.21.15.27:0/2215490453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe150110570 con 0x7fe1500fca50 2024-04-16T05:32:38.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.400+0000 7fe157a00700 1 -- 172.21.15.27:0/2215490453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe150102f60 con 0x7fe1500fca50 2024-04-16T05:32:38.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.400+0000 7fe14cc00700 1 -- 172.21.15.27:0/2215490453 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7fe144013a80 con 0x7fe1500fca50 2024-04-16T05:32:38.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.401+0000 7fe14cc00700 1 --2- 172.21.15.27:0/2215490453 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fe138036880 0x7fe138038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:38.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.401+0000 7fe14cc00700 1 -- 172.21.15.27:0/2215490453 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe144049ac0 con 0x7fe1500fca50 2024-04-16T05:32:38.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.401+0000 7fe14fe00700 1 --2- 172.21.15.27:0/2215490453 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fe138036880 0x7fe138038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:38.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.402+0000 7fe14fe00700 1 --2- 172.21.15.27:0/2215490453 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fe138036880 0x7fe138038d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe13c009990 tx=0x7fe13c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:38.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.407+0000 7fe14cc00700 1 -- 172.21.15.27:0/2215490453 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fe144011960 con 0x7fe1500fca50 2024-04-16T05:32:38.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.631+0000 7fe157a00700 1 -- 172.21.15.27:0/2215490453 --> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fe15005f690 con 0x7fe138036880 2024-04-16T05:32:38.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.635+0000 7fe14cc00700 1 -- 172.21.15.27:0/2215490453 <== mgr.14162 v2:172.21.15.27:6800/2700787222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe15005f690 con 0x7fe138036880 2024-04-16T05:32:38.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.638+0000 7fe157a00700 1 -- 172.21.15.27:0/2215490453 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fe138036880 msgr2=0x7fe138038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:38.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.638+0000 7fe157a00700 1 --2- 172.21.15.27:0/2215490453 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fe138036880 0x7fe138038d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe13c009990 tx=0x7fe13c006e40 comp rx=0 tx=0).stop 2024-04-16T05:32:38.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.638+0000 7fe157a00700 1 -- 172.21.15.27:0/2215490453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1500fca50 msgr2=0x7fe15010f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:38.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.638+0000 7fe157a00700 1 --2- 172.21.15.27:0/2215490453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1500fca50 0x7fe15010f2c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe1440096b0 tx=0x7fe144009e30 comp rx=0 tx=0).stop 2024-04-16T05:32:38.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.638+0000 7fe157a00700 1 -- 172.21.15.27:0/2215490453 shutdown_connections 2024-04-16T05:32:38.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.638+0000 7fe157a00700 1 --2- 172.21.15.27:0/2215490453 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fe138036880 0x7fe138038d30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:38.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.638+0000 7fe157a00700 1 --2- 172.21.15.27:0/2215490453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe1500fca50 0x7fe15010f2c0 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe1440096b0 tx=0x7fe144009e30 comp rx=0 tx=0).stop 2024-04-16T05:32:38.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.638+0000 7fe157a00700 1 -- 172.21.15.27:0/2215490453 >> 172.21.15.27:0/2215490453 conn(0x7fe1500f81d0 msgr2=0x7fe1500fa5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:38.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.638+0000 7fe157a00700 1 -- 172.21.15.27:0/2215490453 shutdown_connections 2024-04-16T05:32:38.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:38.639+0000 7fe157a00700 1 -- 172.21.15.27:0/2215490453 wait complete. 2024-04-16T05:32:38.664 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:38 smithi027 ceph-mon[29206]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:32:39.684 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi122 2024-04-16T05:32:39.684 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-16T05:32:39.684 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/etc/ceph/ceph.conf 2024-04-16T05:32:39.741 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-16T05:32:39.741 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-16T05:32:39.834 INFO:tasks.cephadm:Adding host smithi122 to orchestrator... 2024-04-16T05:32:39.835 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph orch host add smithi122 2024-04-16T05:32:39.907 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:39 smithi027 ceph-mon[29206]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:32:39.907 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:39 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:40.476 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:32:40.963 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:40 smithi027 ceph-mon[29206]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:32:40.964 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:40 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:40.964 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:40 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:40.964 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:40 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:40.964 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:40 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:41.757 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:41 smithi027 ceph-mon[29206]: Deploying daemon node-exporter.smithi027 on smithi027 2024-04-16T05:32:42.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.579+0000 7f2bf2e00700 1 -- 172.21.15.27:0/1389357809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bec071e40 msgr2=0x7f2bec070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:42.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.579+0000 7f2bf2e00700 1 --2- 172.21.15.27:0/1389357809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bec071e40 0x7f2bec070350 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc0048c0 tx=0x7f2bdc009e30 comp rx=0 tx=0).stop 2024-04-16T05:32:42.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.579+0000 7f2bf2e00700 1 -- 172.21.15.27:0/1389357809 shutdown_connections 2024-04-16T05:32:42.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.579+0000 7f2bf2e00700 1 --2- 172.21.15.27:0/1389357809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bec071e40 0x7f2bec070350 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:42.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.579+0000 7f2bf2e00700 1 -- 172.21.15.27:0/1389357809 >> 172.21.15.27:0/1389357809 conn(0x7f2bec0f81c0 msgr2=0x7f2bec0fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:42.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.579+0000 7f2bf2e00700 1 -- 172.21.15.27:0/1389357809 shutdown_connections 2024-04-16T05:32:42.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.579+0000 7f2bf2e00700 1 -- 172.21.15.27:0/1389357809 wait complete. 2024-04-16T05:32:42.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.580+0000 7f2bf2e00700 1 Processor -- start 2024-04-16T05:32:42.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.580+0000 7f2bf2e00700 1 -- start start 2024-04-16T05:32:42.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.581+0000 7f2bf2e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bec071e40 0x7f2bec113710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:42.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.581+0000 7f2bf2e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bec113c50 con 0x7f2bec071e40 2024-04-16T05:32:42.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.581+0000 7f2bebe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bec071e40 0x7f2bec113710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:42.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.581+0000 7f2bebe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bec071e40 0x7f2bec113710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44412/0 (socket says 172.21.15.27:44412) 2024-04-16T05:32:42.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.581+0000 7f2bebe00700 1 -- 172.21.15.27:0/3392059098 learned_addr learned my addr 172.21.15.27:0/3392059098 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:42.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.582+0000 7f2bebe00700 1 -- 172.21.15.27:0/3392059098 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bdc005040 con 0x7f2bec071e40 2024-04-16T05:32:42.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.582+0000 7f2bebe00700 1 --2- 172.21.15.27:0/3392059098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bec071e40 0x7f2bec113710 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc00b900 tx=0x7f2bdc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:42.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.582+0000 7f2be3e00700 1 -- 172.21.15.27:0/3392059098 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bdc00b5c0 con 0x7f2bec071e40 2024-04-16T05:32:42.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.582+0000 7f2bf2e00700 1 -- 172.21.15.27:0/3392059098 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bec113e50 con 0x7f2bec071e40 2024-04-16T05:32:42.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.582+0000 7f2be3e00700 1 -- 172.21.15.27:0/3392059098 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2bdc00be70 con 0x7f2bec071e40 2024-04-16T05:32:42.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.582+0000 7f2be3e00700 1 -- 172.21.15.27:0/3392059098 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bdc01c590 con 0x7f2bec071e40 2024-04-16T05:32:42.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.582+0000 7f2bf2e00700 1 -- 172.21.15.27:0/3392059098 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bec1149c0 con 0x7f2bec071e40 2024-04-16T05:32:42.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.584+0000 7f2bf2e00700 1 -- 172.21.15.27:0/3392059098 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bec105180 con 0x7f2bec071e40 2024-04-16T05:32:42.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.584+0000 7f2be3e00700 1 -- 172.21.15.27:0/3392059098 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f2bdc01c850 con 0x7f2bec071e40 2024-04-16T05:32:42.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.585+0000 7f2be3e00700 1 --2- 172.21.15.27:0/3392059098 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f2bd00368d0 0x7f2bd0038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:42.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.585+0000 7f2be3e00700 1 -- 172.21.15.27:0/3392059098 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2bdc049510 con 0x7f2bec071e40 2024-04-16T05:32:42.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.585+0000 7f2beb400700 1 --2- 172.21.15.27:0/3392059098 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f2bd00368d0 0x7f2bd0038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:42.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.586+0000 7f2beb400700 1 --2- 172.21.15.27:0/3392059098 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f2bd00368d0 0x7f2bd0038d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8009990 tx=0x7f2bd8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:42.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.589+0000 7f2be3e00700 1 -- 172.21.15.27:0/3392059098 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f2bdc011960 con 0x7f2bec071e40 2024-04-16T05:32:42.797 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:42 smithi027 ceph-mon[29206]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:32:42.797 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:42 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:42.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:42.859+0000 7f2bf2e00700 1 -- 172.21.15.27:0/3392059098 --> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi122", "target": ["mon-mgr", ""]}) v1 -- 0x7f2bec114c00 con 0x7f2bd00368d0 2024-04-16T05:32:44.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:43 smithi027 ceph-mon[29206]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi122", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:32:44.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:43 smithi027 ceph-mon[29206]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:32:44.979 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:44 smithi027 ceph-mon[29206]: Deploying cephadm binary to smithi122 2024-04-16T05:32:46.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:45 smithi027 ceph-mon[29206]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:32:47.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:47.225+0000 7f2be3e00700 1 -- 172.21.15.27:0/3392059098 <== mgr.14162 v2:172.21.15.27:6800/2700787222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f2bec114c00 con 0x7f2bd00368d0 2024-04-16T05:32:47.226 INFO:teuthology.orchestra.run.smithi027.stdout:Added host 'smithi122' with addr '172.21.15.122' 2024-04-16T05:32:47.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:47.229+0000 7f2bf2e00700 1 -- 172.21.15.27:0/3392059098 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f2bd00368d0 msgr2=0x7f2bd0038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:47.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:47.229+0000 7f2bf2e00700 1 --2- 172.21.15.27:0/3392059098 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f2bd00368d0 0x7f2bd0038d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8009990 tx=0x7f2bd8006e40 comp rx=0 tx=0).stop 2024-04-16T05:32:47.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:47.230+0000 7f2bf2e00700 1 -- 172.21.15.27:0/3392059098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bec071e40 msgr2=0x7f2bec113710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:47.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:47.230+0000 7f2bf2e00700 1 --2- 172.21.15.27:0/3392059098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bec071e40 0x7f2bec113710 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc00b900 tx=0x7f2bdc009e30 comp rx=0 tx=0).stop 2024-04-16T05:32:47.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:47.230+0000 7f2bf2e00700 1 -- 172.21.15.27:0/3392059098 shutdown_connections 2024-04-16T05:32:47.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:47.230+0000 7f2bf2e00700 1 --2- 172.21.15.27:0/3392059098 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f2bd00368d0 0x7f2bd0038d80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:47.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:47.230+0000 7f2bf2e00700 1 --2- 172.21.15.27:0/3392059098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bec071e40 0x7f2bec113710 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:47.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:47.230+0000 7f2bf2e00700 1 -- 172.21.15.27:0/3392059098 >> 172.21.15.27:0/3392059098 conn(0x7f2bec0f81c0 msgr2=0x7f2bec0fc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:47.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:47.230+0000 7f2bf2e00700 1 -- 172.21.15.27:0/3392059098 shutdown_connections 2024-04-16T05:32:47.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:47.230+0000 7f2bf2e00700 1 -- 172.21.15.27:0/3392059098 wait complete. 2024-04-16T05:32:47.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:47 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:47.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:47 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:47.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:47 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:47.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:47 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:47.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:47 smithi027 ceph-mon[29206]: Deploying daemon alertmanager.smithi027 on smithi027 2024-04-16T05:32:48.533 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:48 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:48.533 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:48 smithi027 ceph-mon[29206]: Added host smithi122 2024-04-16T05:32:48.533 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:48 smithi027 ceph-mon[29206]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:32:48.533 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:48 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:32:48.597 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph orch host ls --format=json 2024-04-16T05:32:49.149 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:32:51.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:50 smithi027 ceph-mon[29206]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:32:52.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.319+0000 7f3038c00700 1 -- 172.21.15.27:0/1497412642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30340fc890 msgr2=0x7f30340fcca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:52.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.319+0000 7f3038c00700 1 --2- 172.21.15.27:0/1497412642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30340fc890 0x7f30340fcca0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3024009c60 tx=0x7f3024004d10 comp rx=0 tx=0).stop 2024-04-16T05:32:52.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.320+0000 7f3038c00700 1 -- 172.21.15.27:0/1497412642 shutdown_connections 2024-04-16T05:32:52.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.320+0000 7f3038c00700 1 --2- 172.21.15.27:0/1497412642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30340fc890 0x7f30340fcca0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:52.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.320+0000 7f3038c00700 1 -- 172.21.15.27:0/1497412642 >> 172.21.15.27:0/1497412642 conn(0x7f30340f7ff0 msgr2=0x7f30340fa420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:52.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.320+0000 7f3038c00700 1 -- 172.21.15.27:0/1497412642 shutdown_connections 2024-04-16T05:32:52.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.320+0000 7f3038c00700 1 -- 172.21.15.27:0/1497412642 wait complete. 2024-04-16T05:32:52.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.321+0000 7f3038c00700 1 Processor -- start 2024-04-16T05:32:52.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.321+0000 7f3038c00700 1 -- start start 2024-04-16T05:32:52.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.321+0000 7f3038c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30340fc890 0x7f303410cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:52.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.321+0000 7f3038c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303410d400 con 0x7f30340fc890 2024-04-16T05:32:52.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.321+0000 7f3033400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30340fc890 0x7f303410cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:52.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.321+0000 7f3033400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30340fc890 0x7f303410cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55604/0 (socket says 172.21.15.27:55604) 2024-04-16T05:32:52.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.321+0000 7f3033400700 1 -- 172.21.15.27:0/657377605 learned_addr learned my addr 172.21.15.27:0/657377605 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:52.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.322+0000 7f3033400700 1 -- 172.21.15.27:0/657377605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3024009940 con 0x7f30340fc890 2024-04-16T05:32:52.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.322+0000 7f3033400700 1 --2- 172.21.15.27:0/657377605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30340fc890 0x7f303410cec0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3024004e60 tx=0x7f3024004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:52.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.322+0000 7f302be00700 1 -- 172.21.15.27:0/657377605 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3024012410 con 0x7f30340fc890 2024-04-16T05:32:52.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.322+0000 7f302be00700 1 -- 172.21.15.27:0/657377605 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3024012a40 con 0x7f30340fc890 2024-04-16T05:32:52.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.322+0000 7f302be00700 1 -- 172.21.15.27:0/657377605 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f302401ba40 con 0x7f30340fc890 2024-04-16T05:32:52.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.322+0000 7f3038c00700 1 -- 172.21.15.27:0/657377605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f303410d600 con 0x7f30340fc890 2024-04-16T05:32:52.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.323+0000 7f3038c00700 1 -- 172.21.15.27:0/657377605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f303410e170 con 0x7f30340fc890 2024-04-16T05:32:52.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.324+0000 7f302be00700 1 -- 172.21.15.27:0/657377605 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f3024019040 con 0x7f30340fc890 2024-04-16T05:32:52.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.324+0000 7f3038c00700 1 -- 172.21.15.27:0/657377605 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3034102da0 con 0x7f30340fc890 2024-04-16T05:32:52.612 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:32:52.613 INFO:teuthology.orchestra.run.smithi027.stdout:[{"addr": "172.21.15.27", "hostname": "smithi027", "labels": [], "status": ""}, {"addr": "172.21.15.122", "hostname": "smithi122", "labels": [], "status": ""}] 2024-04-16T05:32:52.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.324+0000 7f302be00700 1 --2- 172.21.15.27:0/657377605 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f3018036880 0x7f3018038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:52.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.324+0000 7f302be00700 1 -- 172.21.15.27:0/657377605 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3024049c10 con 0x7f30340fc890 2024-04-16T05:32:52.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.325+0000 7f302aa00700 1 --2- 172.21.15.27:0/657377605 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f3018036880 0x7f3018038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:52.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.325+0000 7f302aa00700 1 --2- 172.21.15.27:0/657377605 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f3018036880 0x7f3018038d30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f301c009990 tx=0x7f301c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:52.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.330+0000 7f302be00700 1 -- 172.21.15.27:0/657377605 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f302401b3f0 con 0x7f30340fc890 2024-04-16T05:32:52.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.554+0000 7f3038c00700 1 -- 172.21.15.27:0/657377605 --> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f303405f690 con 0x7f3018036880 2024-04-16T05:32:52.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.556+0000 7f302be00700 1 -- 172.21.15.27:0/657377605 <== mgr.14162 v2:172.21.15.27:6800/2700787222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f303405f690 con 0x7f3018036880 2024-04-16T05:32:52.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.558+0000 7f3038c00700 1 -- 172.21.15.27:0/657377605 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f3018036880 msgr2=0x7f3018038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:52.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.558+0000 7f3038c00700 1 --2- 172.21.15.27:0/657377605 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f3018036880 0x7f3018038d30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f301c009990 tx=0x7f301c006e40 comp rx=0 tx=0).stop 2024-04-16T05:32:52.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.558+0000 7f3038c00700 1 -- 172.21.15.27:0/657377605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30340fc890 msgr2=0x7f303410cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:52.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.558+0000 7f3038c00700 1 --2- 172.21.15.27:0/657377605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30340fc890 0x7f303410cec0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3024004e60 tx=0x7f3024004d10 comp rx=0 tx=0).stop 2024-04-16T05:32:52.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.558+0000 7f3038c00700 1 -- 172.21.15.27:0/657377605 shutdown_connections 2024-04-16T05:32:52.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.558+0000 7f3038c00700 1 --2- 172.21.15.27:0/657377605 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f3018036880 0x7f3018038d30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:52.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.558+0000 7f3038c00700 1 --2- 172.21.15.27:0/657377605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30340fc890 0x7f303410cec0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:52.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.558+0000 7f3038c00700 1 -- 172.21.15.27:0/657377605 >> 172.21.15.27:0/657377605 conn(0x7f30340f7ff0 msgr2=0x7f30341039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:52.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.559+0000 7f3038c00700 1 -- 172.21.15.27:0/657377605 shutdown_connections 2024-04-16T05:32:52.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:52.559+0000 7f3038c00700 1 -- 172.21.15.27:0/657377605 wait complete. 2024-04-16T05:32:53.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:52 smithi027 ceph-mon[29206]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:32:53.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:53 smithi027 ceph-mon[29206]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-16T05:32:54.379 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-16T05:32:54.379 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph osd crush tunables default 2024-04-16T05:32:54.902 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:54 smithi027 ceph-mon[29206]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:32:55.920 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:32:56.660 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:56 smithi027 ceph-mon[29206]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:32:58.687 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:32:58 smithi027 ceph-mon[29206]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:32:59.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.320+0000 7f4d9f800700 1 -- 172.21.15.27:0/3345133295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d98071ea0 msgr2=0x7f4d980703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:32:59.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.320+0000 7f4d9f800700 1 --2- 172.21.15.27:0/3345133295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d98071ea0 0x7f4d980703b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c0048c0 tx=0x7f4d8c009e30 comp rx=0 tx=0).stop 2024-04-16T05:32:59.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.320+0000 7f4d9f800700 1 -- 172.21.15.27:0/3345133295 shutdown_connections 2024-04-16T05:32:59.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.320+0000 7f4d9f800700 1 --2- 172.21.15.27:0/3345133295 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d98071ea0 0x7f4d980703b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:32:59.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.320+0000 7f4d9f800700 1 -- 172.21.15.27:0/3345133295 >> 172.21.15.27:0/3345133295 conn(0x7f4d980f81e0 msgr2=0x7f4d980fa610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:32:59.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.320+0000 7f4d9f800700 1 -- 172.21.15.27:0/3345133295 shutdown_connections 2024-04-16T05:32:59.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.320+0000 7f4d9f800700 1 -- 172.21.15.27:0/3345133295 wait complete. 2024-04-16T05:32:59.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.322+0000 7f4d9f800700 1 Processor -- start 2024-04-16T05:32:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.322+0000 7f4d9f800700 1 -- start start 2024-04-16T05:32:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.322+0000 7f4d9f800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d98071ea0 0x7f4d9810f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.322+0000 7f4d9f800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9810f830 con 0x7f4d98071ea0 2024-04-16T05:32:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.323+0000 7f4d9ce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d98071ea0 0x7f4d9810f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.323+0000 7f4d9ce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d98071ea0 0x7f4d9810f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44406/0 (socket says 172.21.15.27:44406) 2024-04-16T05:32:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.323+0000 7f4d9ce00700 1 -- 172.21.15.27:0/1760353128 learned_addr learned my addr 172.21.15.27:0/1760353128 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:32:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.323+0000 7f4d9ce00700 1 -- 172.21.15.27:0/1760353128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8c005040 con 0x7f4d98071ea0 2024-04-16T05:32:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.323+0000 7f4d9ce00700 1 --2- 172.21.15.27:0/1760353128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d98071ea0 0x7f4d9810f2f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c00b900 tx=0x7f4d8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.324+0000 7f4d94c00700 1 -- 172.21.15.27:0/1760353128 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d8c00b5c0 con 0x7f4d98071ea0 2024-04-16T05:32:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.324+0000 7f4d94c00700 1 -- 172.21.15.27:0/1760353128 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d8c00be70 con 0x7f4d98071ea0 2024-04-16T05:32:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.324+0000 7f4d94c00700 1 -- 172.21.15.27:0/1760353128 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d8c01c590 con 0x7f4d98071ea0 2024-04-16T05:32:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.324+0000 7f4d9f800700 1 -- 172.21.15.27:0/1760353128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d9810fa30 con 0x7f4d98071ea0 2024-04-16T05:32:59.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.324+0000 7f4d9f800700 1 -- 172.21.15.27:0/1760353128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d981105a0 con 0x7f4d98071ea0 2024-04-16T05:32:59.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.325+0000 7f4d94c00700 1 -- 172.21.15.27:0/1760353128 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f4d8c004d30 con 0x7f4d98071ea0 2024-04-16T05:32:59.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.325+0000 7f4d9f800700 1 -- 172.21.15.27:0/1760353128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d981051a0 con 0x7f4d98071ea0 2024-04-16T05:32:59.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.325+0000 7f4d94c00700 1 --2- 172.21.15.27:0/1760353128 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f4d800368d0 0x7f4d80038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:32:59.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.326+0000 7f4d94c00700 1 -- 172.21.15.27:0/1760353128 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4d8c049380 con 0x7f4d98071ea0 2024-04-16T05:32:59.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.326+0000 7f4d97e00700 1 --2- 172.21.15.27:0/1760353128 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f4d800368d0 0x7f4d80038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:32:59.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.327+0000 7f4d97e00700 1 --2- 172.21.15.27:0/1760353128 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f4d800368d0 0x7f4d80038d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4d84009990 tx=0x7f4d84006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:32:59.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.332+0000 7f4d94c00700 1 -- 172.21.15.27:0/1760353128 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f4d8c01b350 con 0x7f4d98071ea0 2024-04-16T05:32:59.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:32:59.565+0000 7f4d9f800700 1 -- 172.21.15.27:0/1760353128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f4d9805f5f0 con 0x7f4d98071ea0 2024-04-16T05:33:00.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:33:00.535+0000 7f4d94c00700 1 -- 172.21.15.27:0/1760353128 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f4d8c021030 con 0x7f4d98071ea0 2024-04-16T05:33:00.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:33:00.540+0000 7f4d9f800700 1 -- 172.21.15.27:0/1760353128 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f4d800368d0 msgr2=0x7f4d80038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:00.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:33:00.540+0000 7f4d9f800700 1 --2- 172.21.15.27:0/1760353128 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f4d800368d0 0x7f4d80038d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4d84009990 tx=0x7f4d84006e40 comp rx=0 tx=0).stop 2024-04-16T05:33:00.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:33:00.540+0000 7f4d9f800700 1 -- 172.21.15.27:0/1760353128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d98071ea0 msgr2=0x7f4d9810f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:00.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:33:00.540+0000 7f4d9f800700 1 --2- 172.21.15.27:0/1760353128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d98071ea0 0x7f4d9810f2f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4d8c00b900 tx=0x7f4d8c009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:00.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:33:00.540+0000 7f4d9f800700 1 -- 172.21.15.27:0/1760353128 shutdown_connections 2024-04-16T05:33:00.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:33:00.540+0000 7f4d9f800700 1 --2- 172.21.15.27:0/1760353128 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f4d800368d0 0x7f4d80038d80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:00.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:33:00.540+0000 7f4d9f800700 1 --2- 172.21.15.27:0/1760353128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4d98071ea0 0x7f4d9810f2f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:00.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:33:00.540+0000 7f4d9f800700 1 -- 172.21.15.27:0/1760353128 >> 172.21.15.27:0/1760353128 conn(0x7f4d980f81e0 msgr2=0x7f4d98105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:00.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:33:00.540+0000 7f4d9f800700 1 -- 172.21.15.27:0/1760353128 shutdown_connections 2024-04-16T05:33:00.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:33:00.540+0000 7f4d9f800700 1 -- 172.21.15.27:0/1760353128 wait complete. 2024-04-16T05:33:00.541 INFO:teuthology.orchestra.run.smithi027.stderr:adjusted tunables profile to default 2024-04-16T05:33:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:00 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:00 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:00 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:00 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:00 smithi027 ceph-mon[29206]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:33:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:00 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1760353128' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-16T05:33:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:00 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:00 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:00.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:00 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-16T05:33:00.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:00 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:01.424 INFO:tasks.cephadm:Adding mon.smithi027 on smithi027 2024-04-16T05:33:01.424 INFO:tasks.cephadm:Adding mon.smithi122 on smithi122 2024-04-16T05:33:01.425 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph orch apply mon '2;smithi027:172.21.15.27=smithi027;smithi122:172.21.15.122=smithi122' 2024-04-16T05:33:01.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:01 smithi027 ceph-mon[29206]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-16T05:33:01.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:01 smithi027 ceph-mon[29206]: Deploying daemon grafana.smithi027 on smithi027 2024-04-16T05:33:01.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:01 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1760353128' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-16T05:33:01.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:01 smithi027 ceph-mon[29206]: osdmap e4: 0 total, 0 up, 0 in 2024-04-16T05:33:01.872 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:01.976 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:02.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:02 smithi027 ceph-mon[29206]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:33:04.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:03 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:04.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.084+0000 7f05e7000700 1 -- 172.21.15.122:0/759346527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05e0103080 msgr2=0x7f05e0103490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:04.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.085+0000 7f05e7000700 1 --2- 172.21.15.122:0/759346527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05e0103080 0x7f05e0103490 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f05d00048c0 tx=0x7f05d0009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:04.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.085+0000 7f05e7000700 1 -- 172.21.15.122:0/759346527 shutdown_connections 2024-04-16T05:33:04.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.085+0000 7f05e7000700 1 --2- 172.21.15.122:0/759346527 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05e0103080 0x7f05e0103490 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:04.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.085+0000 7f05e7000700 1 -- 172.21.15.122:0/759346527 >> 172.21.15.122:0/759346527 conn(0x7f05e0072300 msgr2=0x7f05e0074710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:04.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.085+0000 7f05e7000700 1 -- 172.21.15.122:0/759346527 shutdown_connections 2024-04-16T05:33:04.084 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.085+0000 7f05e7000700 1 -- 172.21.15.122:0/759346527 wait complete. 2024-04-16T05:33:04.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.086+0000 7f05e7000700 1 Processor -- start 2024-04-16T05:33:04.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.086+0000 7f05e7000700 1 -- start start 2024-04-16T05:33:04.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.086+0000 7f05e7000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05e0103080 0x7f05e0113680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:04.085 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.086+0000 7f05e7000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05e0113bc0 con 0x7f05e0103080 2024-04-16T05:33:04.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.087+0000 7f05dfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05e0103080 0x7f05e0113680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:04.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.087+0000 7f05dfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05e0103080 0x7f05e0113680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:51428/0 (socket says 172.21.15.122:51428) 2024-04-16T05:33:04.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.087+0000 7f05dfe00700 1 -- 172.21.15.122:0/3512126353 learned_addr learned my addr 172.21.15.122:0/3512126353 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:33:04.086 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.088+0000 7f05dfe00700 1 -- 172.21.15.122:0/3512126353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d0005040 con 0x7f05e0103080 2024-04-16T05:33:04.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.088+0000 7f05dfe00700 1 --2- 172.21.15.122:0/3512126353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05e0103080 0x7f05e0113680 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009680 tx=0x7f05d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:04.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.089+0000 7f05d7e00700 1 -- 172.21.15.122:0/3512126353 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05d000b5c0 con 0x7f05e0103080 2024-04-16T05:33:04.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.089+0000 7f05e7000700 1 -- 172.21.15.122:0/3512126353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05e0113dc0 con 0x7f05e0103080 2024-04-16T05:33:04.087 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.089+0000 7f05e7000700 1 -- 172.21.15.122:0/3512126353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05e0114930 con 0x7f05e0103080 2024-04-16T05:33:04.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.089+0000 7f05d7e00700 1 -- 172.21.15.122:0/3512126353 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f05d000be70 con 0x7f05e0103080 2024-04-16T05:33:04.088 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.089+0000 7f05d7e00700 1 -- 172.21.15.122:0/3512126353 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05d0013420 con 0x7f05e0103080 2024-04-16T05:33:04.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.090+0000 7f05e7000700 1 -- 172.21.15.122:0/3512126353 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c00052e0 con 0x7f05e0103080 2024-04-16T05:33:04.089 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.090+0000 7f05d7e00700 1 -- 172.21.15.122:0/3512126353 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f05d001c430 con 0x7f05e0103080 2024-04-16T05:33:04.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.091+0000 7f05d7e00700 1 --2- 172.21.15.122:0/3512126353 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f05c4036880 0x7f05c4038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:04.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.091+0000 7f05d7e00700 1 -- 172.21.15.122:0/3512126353 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f05d00494d0 con 0x7f05e0103080 2024-04-16T05:33:04.090 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.092+0000 7f05df400700 1 --2- 172.21.15.122:0/3512126353 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f05c4036880 0x7f05c4038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:04.091 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.092+0000 7f05df400700 1 --2- 172.21.15.122:0/3512126353 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f05c4036880 0x7f05c4038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f05cc009990 tx=0x7f05cc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:04.119 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.120+0000 7f05d7e00700 1 -- 172.21.15.122:0/3512126353 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f05d001c960 con 0x7f05e0103080 2024-04-16T05:33:04.342 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.343+0000 7f05e7000700 1 -- 172.21.15.122:0/3512126353 --> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi027:172.21.15.27=smithi027;smithi122:172.21.15.122=smithi122", "target": ["mon-mgr", ""]}) v1 -- 0x7f05c0000ca0 con 0x7f05c4036880 2024-04-16T05:33:04.355 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.356+0000 7f05d7e00700 1 -- 172.21.15.122:0/3512126353 <== mgr.14162 v2:172.21.15.27:6800/2700787222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f05c0000ca0 con 0x7f05c4036880 2024-04-16T05:33:04.355 INFO:teuthology.orchestra.run.smithi122.stdout:Scheduled mon update... 2024-04-16T05:33:04.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.359+0000 7f05e7000700 1 -- 172.21.15.122:0/3512126353 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f05c4036880 msgr2=0x7f05c4038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:04.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.359+0000 7f05e7000700 1 --2- 172.21.15.122:0/3512126353 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f05c4036880 0x7f05c4038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f05cc009990 tx=0x7f05cc006e40 comp rx=0 tx=0).stop 2024-04-16T05:33:04.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.359+0000 7f05e7000700 1 -- 172.21.15.122:0/3512126353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05e0103080 msgr2=0x7f05e0113680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:04.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.359+0000 7f05e7000700 1 --2- 172.21.15.122:0/3512126353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05e0103080 0x7f05e0113680 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009680 tx=0x7f05d0009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:04.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.360+0000 7f05e7000700 1 -- 172.21.15.122:0/3512126353 shutdown_connections 2024-04-16T05:33:04.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.360+0000 7f05e7000700 1 --2- 172.21.15.122:0/3512126353 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f05c4036880 0x7f05c4038d30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:04.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.360+0000 7f05e7000700 1 --2- 172.21.15.122:0/3512126353 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f05e0103080 0x7f05e0113680 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:04.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.360+0000 7f05e7000700 1 -- 172.21.15.122:0/3512126353 >> 172.21.15.122:0/3512126353 conn(0x7f05e0072300 msgr2=0x7f05e0074210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:04.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.360+0000 7f05e7000700 1 -- 172.21.15.122:0/3512126353 shutdown_connections 2024-04-16T05:33:04.359 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:04.360+0000 7f05e7000700 1 -- 172.21.15.122:0/3512126353 wait complete. 2024-04-16T05:33:05.018 DEBUG:teuthology.orchestra.run.smithi122:mon.smithi122> sudo journalctl -f -n 0 -u ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@mon.smithi122.service 2024-04-16T05:33:05.023 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T05:33:05.023 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T05:33:05.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:04 smithi027 ceph-mon[29206]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:33:05.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:04 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:05.472 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:05.567 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:06.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:05 smithi027 ceph-mon[29206]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi027:172.21.15.27=smithi027;smithi122:172.21.15.122=smithi122", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:33:06.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:05 smithi027 ceph-mon[29206]: Saving service mon spec with placement smithi027:172.21.15.27=smithi027;smithi122:172.21.15.122=smithi122;count:2 2024-04-16T05:33:07.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:06 smithi027 ceph-mon[29206]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:33:07.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.437+0000 7f4155200700 1 -- 172.21.15.122:0/3518616507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4150103080 msgr2=0x7f4150103490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:07.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.437+0000 7f4155200700 1 --2- 172.21.15.122:0/3518616507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4150103080 0x7f4150103490 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f41400048c0 tx=0x7f4140009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:07.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.438+0000 7f4155200700 1 -- 172.21.15.122:0/3518616507 shutdown_connections 2024-04-16T05:33:07.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.438+0000 7f4155200700 1 --2- 172.21.15.122:0/3518616507 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4150103080 0x7f4150103490 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:07.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.438+0000 7f4155200700 1 -- 172.21.15.122:0/3518616507 >> 172.21.15.122:0/3518616507 conn(0x7f4150072300 msgr2=0x7f4150074710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:07.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.438+0000 7f4155200700 1 -- 172.21.15.122:0/3518616507 shutdown_connections 2024-04-16T05:33:07.437 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.438+0000 7f4155200700 1 -- 172.21.15.122:0/3518616507 wait complete. 2024-04-16T05:33:07.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.439+0000 7f4155200700 1 Processor -- start 2024-04-16T05:33:07.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.439+0000 7f4155200700 1 -- start start 2024-04-16T05:33:07.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.439+0000 7f4155200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4150103080 0x7f41501136c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:07.438 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.439+0000 7f4155200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4150113c00 con 0x7f4150103080 2024-04-16T05:33:07.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.440+0000 7f414de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4150103080 0x7f41501136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:07.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.440+0000 7f414de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4150103080 0x7f41501136c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:51452/0 (socket says 172.21.15.122:51452) 2024-04-16T05:33:07.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.440+0000 7f414de00700 1 -- 172.21.15.122:0/2401783421 learned_addr learned my addr 172.21.15.122:0/2401783421 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:33:07.439 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.441+0000 7f414de00700 1 -- 172.21.15.122:0/2401783421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4140005040 con 0x7f4150103080 2024-04-16T05:33:07.440 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.442+0000 7f414de00700 1 --2- 172.21.15.122:0/2401783421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4150103080 0x7f41501136c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f41400096b0 tx=0x7f4140009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:07.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.442+0000 7f4146000700 1 -- 172.21.15.122:0/2401783421 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f414000bed0 con 0x7f4150103080 2024-04-16T05:33:07.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.442+0000 7f4155200700 1 -- 172.21.15.122:0/2401783421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4150113e00 con 0x7f4150103080 2024-04-16T05:33:07.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.442+0000 7f4155200700 1 -- 172.21.15.122:0/2401783421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4150114970 con 0x7f4150103080 2024-04-16T05:33:07.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.442+0000 7f4146000700 1 -- 172.21.15.122:0/2401783421 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f414000b5c0 con 0x7f4150103080 2024-04-16T05:33:07.441 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.442+0000 7f4146000700 1 -- 172.21.15.122:0/2401783421 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41400255c0 con 0x7f4150103080 2024-04-16T05:33:07.442 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.443+0000 7f4155200700 1 -- 172.21.15.122:0/2401783421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4150109570 con 0x7f4150103080 2024-04-16T05:33:07.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.444+0000 7f4146000700 1 -- 172.21.15.122:0/2401783421 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f41400139f0 con 0x7f4150103080 2024-04-16T05:33:07.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.445+0000 7f4146000700 1 --2- 172.21.15.122:0/2401783421 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f41340368d0 0x7f4134038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:07.443 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.445+0000 7f4146000700 1 -- 172.21.15.122:0/2401783421 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f414004a070 con 0x7f4150103080 2024-04-16T05:33:07.444 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.446+0000 7f414d400700 1 --2- 172.21.15.122:0/2401783421 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f41340368d0 0x7f4134038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:07.446 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.448+0000 7f414d400700 1 --2- 172.21.15.122:0/2401783421 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f41340368d0 0x7f4134038d80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f413c009990 tx=0x7f413c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:07.459 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.460+0000 7f4146000700 1 -- 172.21.15.122:0/2401783421 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f414002d330 con 0x7f4150103080 2024-04-16T05:33:07.758 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.759+0000 7f4155200700 1 -- 172.21.15.122:0/2401783421 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f415005f5f0 con 0x7f4150103080 2024-04-16T05:33:07.759 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.760+0000 7f4146000700 1 -- 172.21.15.122:0/2401783421 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f4140011b60 con 0x7f4150103080 2024-04-16T05:33:07.759 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:33:07.759 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","modified":"2024-04-16T05:30:56.860193Z","created":"2024-04-16T05:30:56.860193Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T05:33:07.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.763+0000 7f4155200700 1 -- 172.21.15.122:0/2401783421 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f41340368d0 msgr2=0x7f4134038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:07.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.763+0000 7f4155200700 1 --2- 172.21.15.122:0/2401783421 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f41340368d0 0x7f4134038d80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f413c009990 tx=0x7f413c006e40 comp rx=0 tx=0).stop 2024-04-16T05:33:07.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.763+0000 7f4155200700 1 -- 172.21.15.122:0/2401783421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4150103080 msgr2=0x7f41501136c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:07.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.763+0000 7f4155200700 1 --2- 172.21.15.122:0/2401783421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4150103080 0x7f41501136c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f41400096b0 tx=0x7f4140009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:07.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.763+0000 7f4155200700 1 -- 172.21.15.122:0/2401783421 shutdown_connections 2024-04-16T05:33:07.762 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.763+0000 7f4155200700 1 --2- 172.21.15.122:0/2401783421 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f41340368d0 0x7f4134038d80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:07.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.763+0000 7f4155200700 1 --2- 172.21.15.122:0/2401783421 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4150103080 0x7f41501136c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:07.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.763+0000 7f4155200700 1 -- 172.21.15.122:0/2401783421 >> 172.21.15.122:0/2401783421 conn(0x7f4150072300 msgr2=0x7f4150074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:07.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.763+0000 7f4155200700 1 -- 172.21.15.122:0/2401783421 shutdown_connections 2024-04-16T05:33:07.763 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:07.763+0000 7f4155200700 1 -- 172.21.15.122:0/2401783421 wait complete. 2024-04-16T05:33:07.763 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-16T05:33:08.891 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:08 smithi027 ceph-mon[29206]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:33:08.891 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:08 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/2401783421' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:33:09.404 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T05:33:09.405 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T05:33:09.842 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:09.947 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:11.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:10 smithi027 ceph-mon[29206]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:33:11.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.846+0000 7fbea9800700 1 -- 172.21.15.122:0/2032994040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbea40fec70 msgr2=0x7fbea40ff080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:11.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.846+0000 7fbea9800700 1 --2- 172.21.15.122:0/2032994040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbea40fec70 0x7fbea40ff080 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbe940048c0 tx=0x7fbe94009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:11.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.846+0000 7fbea9800700 1 -- 172.21.15.122:0/2032994040 shutdown_connections 2024-04-16T05:33:11.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.846+0000 7fbea9800700 1 --2- 172.21.15.122:0/2032994040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbea40fec70 0x7fbea40ff080 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:11.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.846+0000 7fbea9800700 1 -- 172.21.15.122:0/2032994040 >> 172.21.15.122:0/2032994040 conn(0x7fbea40fa5d0 msgr2=0x7fbea40fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:11.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.846+0000 7fbea9800700 1 -- 172.21.15.122:0/2032994040 shutdown_connections 2024-04-16T05:33:11.845 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.846+0000 7fbea9800700 1 -- 172.21.15.122:0/2032994040 wait complete. 2024-04-16T05:33:11.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.847+0000 7fbea9800700 1 Processor -- start 2024-04-16T05:33:11.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.847+0000 7fbea9800700 1 -- start start 2024-04-16T05:33:11.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.848+0000 7fbea9800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbea40fec70 0x7fbea410f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:11.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.848+0000 7fbea9800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea410f7c0 con 0x7fbea40fec70 2024-04-16T05:33:11.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.848+0000 7fbea2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbea40fec70 0x7fbea410f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:11.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.848+0000 7fbea2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbea40fec70 0x7fbea410f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:40482/0 (socket says 172.21.15.122:40482) 2024-04-16T05:33:11.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.848+0000 7fbea2a00700 1 -- 172.21.15.122:0/602803892 learned_addr learned my addr 172.21.15.122:0/602803892 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:33:11.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.849+0000 7fbea2a00700 1 -- 172.21.15.122:0/602803892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe94005040 con 0x7fbea40fec70 2024-04-16T05:33:11.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.849+0000 7fbea2a00700 1 --2- 172.21.15.122:0/602803892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbea40fec70 0x7fbea410f280 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbe94009680 tx=0x7fbe94009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:11.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.850+0000 7fbe9aa00700 1 -- 172.21.15.122:0/602803892 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe9400b5c0 con 0x7fbea40fec70 2024-04-16T05:33:11.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.850+0000 7fbea9800700 1 -- 172.21.15.122:0/602803892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea410f9c0 con 0x7fbea40fec70 2024-04-16T05:33:11.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.850+0000 7fbea9800700 1 -- 172.21.15.122:0/602803892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea4110530 con 0x7fbea40fec70 2024-04-16T05:33:11.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.850+0000 7fbe9aa00700 1 -- 172.21.15.122:0/602803892 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe9400be70 con 0x7fbea40fec70 2024-04-16T05:33:11.849 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.850+0000 7fbe9aa00700 1 -- 172.21.15.122:0/602803892 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe940135c0 con 0x7fbea40fec70 2024-04-16T05:33:11.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.851+0000 7fbea9800700 1 -- 172.21.15.122:0/602803892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea4105180 con 0x7fbea40fec70 2024-04-16T05:33:11.850 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.852+0000 7fbe9aa00700 1 -- 172.21.15.122:0/602803892 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7fbe94004d30 con 0x7fbea40fec70 2024-04-16T05:33:11.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.852+0000 7fbe9aa00700 1 --2- 172.21.15.122:0/602803892 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fbe88036830 0x7fbe88038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:11.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.852+0000 7fbe9aa00700 1 -- 172.21.15.122:0/602803892 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbe94049410 con 0x7fbea40fec70 2024-04-16T05:33:11.851 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.853+0000 7fbea2000700 1 --2- 172.21.15.122:0/602803892 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fbe88036830 0x7fbe88038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:11.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.854+0000 7fbea2000700 1 --2- 172.21.15.122:0/602803892 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fbe88036830 0x7fbe88038ce0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbe90009990 tx=0x7fbe90006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:11.867 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:11.868+0000 7fbe9aa00700 1 -- 172.21.15.122:0/602803892 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fbe94011550 con 0x7fbea40fec70 2024-04-16T05:33:12.160 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:12.160+0000 7fbea9800700 1 -- 172.21.15.122:0/602803892 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbea405f5f0 con 0x7fbea40fec70 2024-04-16T05:33:12.161 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:12.162+0000 7fbe9aa00700 1 -- 172.21.15.122:0/602803892 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fbe94021020 con 0x7fbea40fec70 2024-04-16T05:33:12.161 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:33:12.161 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","modified":"2024-04-16T05:30:56.860193Z","created":"2024-04-16T05:30:56.860193Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T05:33:12.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:12.165+0000 7fbea9800700 1 -- 172.21.15.122:0/602803892 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fbe88036830 msgr2=0x7fbe88038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:12.164 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:12.165+0000 7fbea9800700 1 --2- 172.21.15.122:0/602803892 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fbe88036830 0x7fbe88038ce0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbe90009990 tx=0x7fbe90006e40 comp rx=0 tx=0).stop 2024-04-16T05:33:12.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:12.165+0000 7fbea9800700 1 -- 172.21.15.122:0/602803892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbea40fec70 msgr2=0x7fbea410f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:12.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:12.165+0000 7fbea9800700 1 --2- 172.21.15.122:0/602803892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbea40fec70 0x7fbea410f280 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbe94009680 tx=0x7fbe94009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:12.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:12.166+0000 7fbea9800700 1 -- 172.21.15.122:0/602803892 shutdown_connections 2024-04-16T05:33:12.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:12.166+0000 7fbea9800700 1 --2- 172.21.15.122:0/602803892 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fbe88036830 0x7fbe88038ce0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:12.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:12.166+0000 7fbea9800700 1 --2- 172.21.15.122:0/602803892 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbea40fec70 0x7fbea410f280 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:12.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:12.166+0000 7fbea9800700 1 -- 172.21.15.122:0/602803892 >> 172.21.15.122:0/602803892 conn(0x7fbea40fa5d0 msgr2=0x7fbea40fb170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:12.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:12.166+0000 7fbea9800700 1 -- 172.21.15.122:0/602803892 shutdown_connections 2024-04-16T05:33:12.165 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:12.166+0000 7fbea9800700 1 -- 172.21.15.122:0/602803892 wait complete. 2024-04-16T05:33:12.165 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-16T05:33:13.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:12 smithi027 ceph-mon[29206]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:33:13.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:12 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/602803892' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:33:13.810 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T05:33:13.810 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T05:33:14.258 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:14.363 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:15.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:14 smithi027 ceph-mon[29206]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:33:16.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.109+0000 7f09f6800700 1 -- 172.21.15.122:0/3502171846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f00fca60 msgr2=0x7f09f00fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:16.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.109+0000 7f09f6800700 1 --2- 172.21.15.122:0/3502171846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f00fca60 0x7f09f00fce70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f09e00048c0 tx=0x7f09e0009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:16.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.110+0000 7f09f6800700 1 -- 172.21.15.122:0/3502171846 shutdown_connections 2024-04-16T05:33:16.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.110+0000 7f09f6800700 1 --2- 172.21.15.122:0/3502171846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f00fca60 0x7f09f00fce70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:16.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.110+0000 7f09f6800700 1 -- 172.21.15.122:0/3502171846 >> 172.21.15.122:0/3502171846 conn(0x7f09f00f81c0 msgr2=0x7f09f00fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:16.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.110+0000 7f09f6800700 1 -- 172.21.15.122:0/3502171846 shutdown_connections 2024-04-16T05:33:16.109 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.110+0000 7f09f6800700 1 -- 172.21.15.122:0/3502171846 wait complete. 2024-04-16T05:33:16.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.111+0000 7f09f6800700 1 Processor -- start 2024-04-16T05:33:16.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.111+0000 7f09f6800700 1 -- start start 2024-04-16T05:33:16.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.112+0000 7f09f6800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f00fca60 0x7f09f010f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:16.110 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.112+0000 7f09f6800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f010f7e0 con 0x7f09f00fca60 2024-04-16T05:33:16.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.113+0000 7f09ef400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f00fca60 0x7f09f010f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:16.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.113+0000 7f09ef400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f00fca60 0x7f09f010f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:40508/0 (socket says 172.21.15.122:40508) 2024-04-16T05:33:16.111 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.113+0000 7f09ef400700 1 -- 172.21.15.122:0/2729697736 learned_addr learned my addr 172.21.15.122:0/2729697736 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:33:16.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.113+0000 7f09ef400700 1 -- 172.21.15.122:0/2729697736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e0005040 con 0x7f09f00fca60 2024-04-16T05:33:16.112 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.114+0000 7f09ef400700 1 --2- 172.21.15.122:0/2729697736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f00fca60 0x7f09f010f2a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f09e00096b0 tx=0x7f09e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:16.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.114+0000 7f09e7400700 1 -- 172.21.15.122:0/2729697736 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09e000bed0 con 0x7f09f00fca60 2024-04-16T05:33:16.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.114+0000 7f09f6800700 1 -- 172.21.15.122:0/2729697736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f010f9e0 con 0x7f09f00fca60 2024-04-16T05:33:16.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.114+0000 7f09e7400700 1 -- 172.21.15.122:0/2729697736 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09e000b5c0 con 0x7f09f00fca60 2024-04-16T05:33:16.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.114+0000 7f09f6800700 1 -- 172.21.15.122:0/2729697736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f0110550 con 0x7f09f00fca60 2024-04-16T05:33:16.113 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.114+0000 7f09e7400700 1 -- 172.21.15.122:0/2729697736 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09e0025560 con 0x7f09f00fca60 2024-04-16T05:33:16.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.115+0000 7f09f6800700 1 -- 172.21.15.122:0/2729697736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f0102f70 con 0x7f09f00fca60 2024-04-16T05:33:16.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.116+0000 7f09e7400700 1 -- 172.21.15.122:0/2729697736 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f09e0004d30 con 0x7f09f00fca60 2024-04-16T05:33:16.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.117+0000 7f09e7400700 1 --2- 172.21.15.122:0/2729697736 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f09d4036880 0x7f09d4038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:16.115 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.117+0000 7f09e7400700 1 -- 172.21.15.122:0/2729697736 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f09e0049f00 con 0x7f09f00fca60 2024-04-16T05:33:16.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.118+0000 7f09eea00700 1 --2- 172.21.15.122:0/2729697736 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f09d4036880 0x7f09d4038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:16.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.119+0000 7f09eea00700 1 --2- 172.21.15.122:0/2729697736 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f09d4036880 0x7f09d4038d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f09dc009990 tx=0x7f09dc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:16.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.130+0000 7f09e7400700 1 -- 172.21.15.122:0/2729697736 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f09e0011550 con 0x7f09f00fca60 2024-04-16T05:33:16.426 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.427+0000 7f09f6800700 1 -- 172.21.15.122:0/2729697736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09f004bed0 con 0x7f09f00fca60 2024-04-16T05:33:16.428 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.429+0000 7f09e7400700 1 -- 172.21.15.122:0/2729697736 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f09e0011370 con 0x7f09f00fca60 2024-04-16T05:33:16.428 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:33:16.428 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","modified":"2024-04-16T05:30:56.860193Z","created":"2024-04-16T05:30:56.860193Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T05:33:16.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.431+0000 7f09f6800700 1 -- 172.21.15.122:0/2729697736 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f09d4036880 msgr2=0x7f09d4038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:16.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.432+0000 7f09f6800700 1 --2- 172.21.15.122:0/2729697736 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f09d4036880 0x7f09d4038d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f09dc009990 tx=0x7f09dc006e40 comp rx=0 tx=0).stop 2024-04-16T05:33:16.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.432+0000 7f09f6800700 1 -- 172.21.15.122:0/2729697736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f00fca60 msgr2=0x7f09f010f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:16.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.432+0000 7f09f6800700 1 --2- 172.21.15.122:0/2729697736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f00fca60 0x7f09f010f2a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f09e00096b0 tx=0x7f09e0009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:16.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.432+0000 7f09f6800700 1 -- 172.21.15.122:0/2729697736 shutdown_connections 2024-04-16T05:33:16.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.432+0000 7f09f6800700 1 --2- 172.21.15.122:0/2729697736 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f09d4036880 0x7f09d4038d30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:16.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.432+0000 7f09f6800700 1 --2- 172.21.15.122:0/2729697736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f00fca60 0x7f09f010f2a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:16.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.432+0000 7f09f6800700 1 -- 172.21.15.122:0/2729697736 >> 172.21.15.122:0/2729697736 conn(0x7f09f00f81c0 msgr2=0x7f09f00fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:16.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.432+0000 7f09f6800700 1 -- 172.21.15.122:0/2729697736 shutdown_connections 2024-04-16T05:33:16.431 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:16.432+0000 7f09f6800700 1 -- 172.21.15.122:0/2729697736 wait complete. 2024-04-16T05:33:16.432 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-16T05:33:17.023 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:16 smithi027 ceph-mon[29206]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:33:17.023 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:16 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/2729697736' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:33:18.133 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T05:33:18.134 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T05:33:18.605 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:18.712 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:19.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:18 smithi027 ceph-mon[29206]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:33:20.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.683+0000 7fde27e00700 1 -- 172.21.15.122:0/1368054691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde28103160 msgr2=0x7fde28103570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:20.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.683+0000 7fde27e00700 1 --2- 172.21.15.122:0/1368054691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde28103160 0x7fde28103570 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fde180048c0 tx=0x7fde18009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:20.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.683+0000 7fde27e00700 1 -- 172.21.15.122:0/1368054691 shutdown_connections 2024-04-16T05:33:20.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.683+0000 7fde27e00700 1 --2- 172.21.15.122:0/1368054691 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde28103160 0x7fde28103570 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:20.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.683+0000 7fde27e00700 1 -- 172.21.15.122:0/1368054691 >> 172.21.15.122:0/1368054691 conn(0x7fde28072080 msgr2=0x7fde28074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:20.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.684+0000 7fde27e00700 1 -- 172.21.15.122:0/1368054691 shutdown_connections 2024-04-16T05:33:20.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.684+0000 7fde27e00700 1 -- 172.21.15.122:0/1368054691 wait complete. 2024-04-16T05:33:20.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.685+0000 7fde27e00700 1 Processor -- start 2024-04-16T05:33:20.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.685+0000 7fde27e00700 1 -- start start 2024-04-16T05:33:20.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.686+0000 7fde27e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde28103160 0x7fde28113490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:20.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.686+0000 7fde27e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde281139d0 con 0x7fde28103160 2024-04-16T05:33:20.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.686+0000 7fde26a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde28103160 0x7fde28113490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:20.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.686+0000 7fde26a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde28103160 0x7fde28113490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:40530/0 (socket says 172.21.15.122:40530) 2024-04-16T05:33:20.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.686+0000 7fde26a00700 1 -- 172.21.15.122:0/2709744442 learned_addr learned my addr 172.21.15.122:0/2709744442 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:33:20.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.686+0000 7fde26a00700 1 -- 172.21.15.122:0/2709744442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde18005040 con 0x7fde28103160 2024-04-16T05:33:20.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.687+0000 7fde26a00700 1 --2- 172.21.15.122:0/2709744442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde28103160 0x7fde28113490 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fde18009b80 tx=0x7fde18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:20.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.687+0000 7fde1ea00700 1 -- 172.21.15.122:0/2709744442 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde1800b5c0 con 0x7fde28103160 2024-04-16T05:33:20.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.687+0000 7fde27e00700 1 -- 172.21.15.122:0/2709744442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde28113bd0 con 0x7fde28103160 2024-04-16T05:33:20.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.687+0000 7fde1ea00700 1 -- 172.21.15.122:0/2709744442 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fde1800be70 con 0x7fde28103160 2024-04-16T05:33:20.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.687+0000 7fde1ea00700 1 -- 172.21.15.122:0/2709744442 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde1801c570 con 0x7fde28103160 2024-04-16T05:33:20.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.687+0000 7fde27e00700 1 -- 172.21.15.122:0/2709744442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde28114740 con 0x7fde28103160 2024-04-16T05:33:20.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.688+0000 7fde27e00700 1 -- 172.21.15.122:0/2709744442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde28109390 con 0x7fde28103160 2024-04-16T05:33:20.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.688+0000 7fde1ea00700 1 -- 172.21.15.122:0/2709744442 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7fde18004d90 con 0x7fde28103160 2024-04-16T05:33:20.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.689+0000 7fde1ea00700 1 --2- 172.21.15.122:0/2709744442 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fde0c0368a0 0x7fde0c038d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:20.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.689+0000 7fde1ea00700 1 -- 172.21.15.122:0/2709744442 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fde1804bbb0 con 0x7fde28103160 2024-04-16T05:33:20.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.689+0000 7fde26000700 1 --2- 172.21.15.122:0/2709744442 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fde0c0368a0 0x7fde0c038d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:20.688 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.690+0000 7fde26000700 1 --2- 172.21.15.122:0/2709744442 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fde0c0368a0 0x7fde0c038d50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fde14009990 tx=0x7fde14006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:20.697 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.698+0000 7fde1ea00700 1 -- 172.21.15.122:0/2709744442 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fde18011960 con 0x7fde28103160 2024-04-16T05:33:20.769 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:20 smithi027 ceph-mon[29206]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:33:20.769 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:20 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:20.769 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:20 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:20.769 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:20 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:20.769 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:20 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:20.769 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:20 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:20.769 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:20 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:20.992 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.993+0000 7fde27e00700 1 -- 172.21.15.122:0/2709744442 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde28115150 con 0x7fde28103160 2024-04-16T05:33:20.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.995+0000 7fde1ea00700 1 -- 172.21.15.122:0/2709744442 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fde18011370 con 0x7fde28103160 2024-04-16T05:33:20.994 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:33:20.994 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","modified":"2024-04-16T05:30:56.860193Z","created":"2024-04-16T05:30:56.860193Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T05:33:20.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.997+0000 7fde27e00700 1 -- 172.21.15.122:0/2709744442 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fde0c0368a0 msgr2=0x7fde0c038d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:20.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.997+0000 7fde27e00700 1 --2- 172.21.15.122:0/2709744442 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fde0c0368a0 0x7fde0c038d50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fde14009990 tx=0x7fde14006e40 comp rx=0 tx=0).stop 2024-04-16T05:33:20.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.997+0000 7fde27e00700 1 -- 172.21.15.122:0/2709744442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde28103160 msgr2=0x7fde28113490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:20.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.997+0000 7fde27e00700 1 --2- 172.21.15.122:0/2709744442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde28103160 0x7fde28113490 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fde18009b80 tx=0x7fde18009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:20.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.997+0000 7fde27e00700 1 -- 172.21.15.122:0/2709744442 shutdown_connections 2024-04-16T05:33:20.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.997+0000 7fde27e00700 1 --2- 172.21.15.122:0/2709744442 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fde0c0368a0 0x7fde0c038d50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:20.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.998+0000 7fde27e00700 1 --2- 172.21.15.122:0/2709744442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde28103160 0x7fde28113490 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:20.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.998+0000 7fde27e00700 1 -- 172.21.15.122:0/2709744442 >> 172.21.15.122:0/2709744442 conn(0x7fde28072080 msgr2=0x7fde28109fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:20.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.998+0000 7fde27e00700 1 -- 172.21.15.122:0/2709744442 shutdown_connections 2024-04-16T05:33:20.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:20.998+0000 7fde27e00700 1 -- 172.21.15.122:0/2709744442 wait complete. 2024-04-16T05:33:20.998 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-16T05:33:22.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:21 smithi027 ceph-mon[29206]: Deploying daemon prometheus.smithi027 on smithi027 2024-04-16T05:33:22.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:21 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/2709744442' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:33:22.717 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T05:33:22.718 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T05:33:23.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:22 smithi027 ceph-mon[29206]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:33:23.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:22 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:23.178 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:23.283 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:25.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:24 smithi027 ceph-mon[29206]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:33:25.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.187+0000 7f487be00700 1 -- 172.21.15.122:0/2737341859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874065d60 msgr2=0x7f4874101a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:25.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.187+0000 7f487be00700 1 --2- 172.21.15.122:0/2737341859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874065d60 0x7f4874101a60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f48680048c0 tx=0x7f4868009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:25.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.187+0000 7f487be00700 1 -- 172.21.15.122:0/2737341859 shutdown_connections 2024-04-16T05:33:25.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.187+0000 7f487be00700 1 --2- 172.21.15.122:0/2737341859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874065d60 0x7f4874101a60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:25.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.187+0000 7f487be00700 1 -- 172.21.15.122:0/2737341859 >> 172.21.15.122:0/2737341859 conn(0x7f48740f70d0 msgr2=0x7f48740f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:25.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.187+0000 7f487be00700 1 -- 172.21.15.122:0/2737341859 shutdown_connections 2024-04-16T05:33:25.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.187+0000 7f487be00700 1 -- 172.21.15.122:0/2737341859 wait complete. 2024-04-16T05:33:25.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.188+0000 7f487be00700 1 Processor -- start 2024-04-16T05:33:25.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.188+0000 7f487be00700 1 -- start start 2024-04-16T05:33:25.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.188+0000 7f487be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874065d60 0x7f487410f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:25.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.188+0000 7f487be00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487410f7d0 con 0x7f4874065d60 2024-04-16T05:33:25.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.189+0000 7f4879400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874065d60 0x7f487410f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:25.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.189+0000 7f4879400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874065d60 0x7f487410f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:47304/0 (socket says 172.21.15.122:47304) 2024-04-16T05:33:25.188 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.189+0000 7f4879400700 1 -- 172.21.15.122:0/2278115624 learned_addr learned my addr 172.21.15.122:0/2278115624 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:33:25.189 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.190+0000 7f4879400700 1 -- 172.21.15.122:0/2278115624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4868005040 con 0x7f4874065d60 2024-04-16T05:33:25.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.191+0000 7f4879400700 1 --2- 172.21.15.122:0/2278115624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874065d60 0x7f487410f290 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4868009680 tx=0x7f4868009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:25.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.192+0000 7f4871600700 1 -- 172.21.15.122:0/2278115624 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f486800bed0 con 0x7f4874065d60 2024-04-16T05:33:25.190 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.192+0000 7f487be00700 1 -- 172.21.15.122:0/2278115624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f487410f9d0 con 0x7f4874065d60 2024-04-16T05:33:25.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.192+0000 7f4871600700 1 -- 172.21.15.122:0/2278115624 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f486800b5c0 con 0x7f4874065d60 2024-04-16T05:33:25.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.192+0000 7f4871600700 1 -- 172.21.15.122:0/2278115624 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f486801c5b0 con 0x7f4874065d60 2024-04-16T05:33:25.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.192+0000 7f487be00700 1 -- 172.21.15.122:0/2278115624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4874110540 con 0x7f4874065d60 2024-04-16T05:33:25.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.193+0000 7f487be00700 1 -- 172.21.15.122:0/2278115624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4874105170 con 0x7f4874065d60 2024-04-16T05:33:25.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.193+0000 7f4871600700 1 -- 172.21.15.122:0/2278115624 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f4868004d30 con 0x7f4874065d60 2024-04-16T05:33:25.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.194+0000 7f4871600700 1 --2- 172.21.15.122:0/2278115624 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f485c036880 0x7f485c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:25.192 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.194+0000 7f4871600700 1 -- 172.21.15.122:0/2278115624 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4868049990 con 0x7f4874065d60 2024-04-16T05:33:25.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.194+0000 7f4878a00700 1 --2- 172.21.15.122:0/2278115624 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f485c036880 0x7f485c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:25.194 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.195+0000 7f4878a00700 1 --2- 172.21.15.122:0/2278115624 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f485c036880 0x7f485c038d30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4860009990 tx=0x7f4860006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:25.209 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.210+0000 7f4871600700 1 -- 172.21.15.122:0/2278115624 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f4868011550 con 0x7f4874065d60 2024-04-16T05:33:25.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.498+0000 7f487be00700 1 -- 172.21.15.122:0/2278115624 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4874110f50 con 0x7f4874065d60 2024-04-16T05:33:25.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.500+0000 7f4871600700 1 -- 172.21.15.122:0/2278115624 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f4868011370 con 0x7f4874065d60 2024-04-16T05:33:25.499 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:33:25.499 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","modified":"2024-04-16T05:30:56.860193Z","created":"2024-04-16T05:30:56.860193Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T05:33:25.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.502+0000 7f487be00700 1 -- 172.21.15.122:0/2278115624 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f485c036880 msgr2=0x7f485c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:25.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.502+0000 7f487be00700 1 --2- 172.21.15.122:0/2278115624 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f485c036880 0x7f485c038d30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4860009990 tx=0x7f4860006e40 comp rx=0 tx=0).stop 2024-04-16T05:33:25.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.502+0000 7f487be00700 1 -- 172.21.15.122:0/2278115624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874065d60 msgr2=0x7f487410f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:25.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.502+0000 7f487be00700 1 --2- 172.21.15.122:0/2278115624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874065d60 0x7f487410f290 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4868009680 tx=0x7f4868009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:25.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.502+0000 7f487be00700 1 -- 172.21.15.122:0/2278115624 shutdown_connections 2024-04-16T05:33:25.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.502+0000 7f487be00700 1 --2- 172.21.15.122:0/2278115624 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f485c036880 0x7f485c038d30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:25.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.502+0000 7f487be00700 1 --2- 172.21.15.122:0/2278115624 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4874065d60 0x7f487410f290 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:25.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.502+0000 7f487be00700 1 -- 172.21.15.122:0/2278115624 >> 172.21.15.122:0/2278115624 conn(0x7f48740f70d0 msgr2=0x7f4874105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:25.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.502+0000 7f487be00700 1 -- 172.21.15.122:0/2278115624 shutdown_connections 2024-04-16T05:33:25.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:25.503+0000 7f487be00700 1 -- 172.21.15.122:0/2278115624 wait complete. 2024-04-16T05:33:25.502 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-16T05:33:26.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:25 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/2278115624' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:33:27.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:26 smithi027 ceph-mon[29206]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:33:27.246 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T05:33:27.247 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T05:33:27.690 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:27.796 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:28.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:27 smithi027 ceph-mon[29206]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:33:29.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.681+0000 7f0fda000700 1 -- 172.21.15.122:0/2066915032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4103080 msgr2=0x7f0fd4103490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:29.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.681+0000 7f0fda000700 1 --2- 172.21.15.122:0/2066915032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4103080 0x7f0fd4103490 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0fc40048c0 tx=0x7f0fc4009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:29.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.681+0000 7f0fda000700 1 -- 172.21.15.122:0/2066915032 shutdown_connections 2024-04-16T05:33:29.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.681+0000 7f0fda000700 1 --2- 172.21.15.122:0/2066915032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4103080 0x7f0fd4103490 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:29.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.681+0000 7f0fda000700 1 -- 172.21.15.122:0/2066915032 >> 172.21.15.122:0/2066915032 conn(0x7f0fd4072300 msgr2=0x7f0fd4074710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:29.680 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.681+0000 7f0fda000700 1 -- 172.21.15.122:0/2066915032 shutdown_connections 2024-04-16T05:33:29.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.681+0000 7f0fda000700 1 -- 172.21.15.122:0/2066915032 wait complete. 2024-04-16T05:33:29.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.682+0000 7f0fda000700 1 Processor -- start 2024-04-16T05:33:29.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.682+0000 7f0fda000700 1 -- start start 2024-04-16T05:33:29.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.682+0000 7f0fda000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4103080 0x7f0fd41136c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:29.681 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.682+0000 7f0fda000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd4113c00 con 0x7f0fd4103080 2024-04-16T05:33:29.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.683+0000 7f0fd3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4103080 0x7f0fd41136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:29.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.683+0000 7f0fd3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4103080 0x7f0fd41136c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:47318/0 (socket says 172.21.15.122:47318) 2024-04-16T05:33:29.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.683+0000 7f0fd3400700 1 -- 172.21.15.122:0/2561747781 learned_addr learned my addr 172.21.15.122:0/2561747781 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:33:29.682 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.684+0000 7f0fd3400700 1 -- 172.21.15.122:0/2561747781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc4005040 con 0x7f0fd4103080 2024-04-16T05:33:29.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.684+0000 7f0fd3400700 1 --2- 172.21.15.122:0/2561747781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4103080 0x7f0fd41136c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4009680 tx=0x7f0fc4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:29.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.684+0000 7f0fcb400700 1 -- 172.21.15.122:0/2561747781 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fc400b5c0 con 0x7f0fd4103080 2024-04-16T05:33:29.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.684+0000 7f0fcb400700 1 -- 172.21.15.122:0/2561747781 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fc400be70 con 0x7f0fd4103080 2024-04-16T05:33:29.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.684+0000 7f0fda000700 1 -- 172.21.15.122:0/2561747781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd4113e00 con 0x7f0fd4103080 2024-04-16T05:33:29.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.685+0000 7f0fcb400700 1 -- 172.21.15.122:0/2561747781 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fc40135a0 con 0x7f0fd4103080 2024-04-16T05:33:29.683 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.685+0000 7f0fda000700 1 -- 172.21.15.122:0/2561747781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd4114970 con 0x7f0fd4103080 2024-04-16T05:33:29.684 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.686+0000 7f0fda000700 1 -- 172.21.15.122:0/2561747781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd4109570 con 0x7f0fd4103080 2024-04-16T05:33:29.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.686+0000 7f0fcb400700 1 -- 172.21.15.122:0/2561747781 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f0fc4004d30 con 0x7f0fd4103080 2024-04-16T05:33:29.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.687+0000 7f0fcb400700 1 --2- 172.21.15.122:0/2561747781 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f0fb8036880 0x7f0fb8038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:29.685 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.687+0000 7f0fcb400700 1 -- 172.21.15.122:0/2561747781 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0fc4049650 con 0x7f0fd4103080 2024-04-16T05:33:29.686 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.687+0000 7f0fd2a00700 1 --2- 172.21.15.122:0/2561747781 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f0fb8036880 0x7f0fb8038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:29.687 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.689+0000 7f0fd2a00700 1 --2- 172.21.15.122:0/2561747781 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f0fb8036880 0x7f0fb8038d30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0009990 tx=0x7f0fc0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:29.700 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.702+0000 7f0fcb400700 1 -- 172.21.15.122:0/2561747781 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f0fc4011960 con 0x7f0fd4103080 2024-04-16T05:33:29.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:29.999+0000 7f0fda000700 1 -- 172.21.15.122:0/2561747781 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0fd4115440 con 0x7f0fd4103080 2024-04-16T05:33:30.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:30.001+0000 7f0fcb400700 1 -- 172.21.15.122:0/2561747781 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f0fc402b300 con 0x7f0fd4103080 2024-04-16T05:33:30.000 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:33:30.000 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","modified":"2024-04-16T05:30:56.860193Z","created":"2024-04-16T05:30:56.860193Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T05:33:30.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:30.004+0000 7f0fda000700 1 -- 172.21.15.122:0/2561747781 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f0fb8036880 msgr2=0x7f0fb8038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:30.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:30.004+0000 7f0fda000700 1 --2- 172.21.15.122:0/2561747781 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f0fb8036880 0x7f0fb8038d30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0009990 tx=0x7f0fc0006e40 comp rx=0 tx=0).stop 2024-04-16T05:33:30.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:30.004+0000 7f0fda000700 1 -- 172.21.15.122:0/2561747781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4103080 msgr2=0x7f0fd41136c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:30.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:30.004+0000 7f0fda000700 1 --2- 172.21.15.122:0/2561747781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4103080 0x7f0fd41136c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4009680 tx=0x7f0fc4009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:30.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:30.004+0000 7f0fda000700 1 -- 172.21.15.122:0/2561747781 shutdown_connections 2024-04-16T05:33:30.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:30.004+0000 7f0fda000700 1 --2- 172.21.15.122:0/2561747781 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f0fb8036880 0x7f0fb8038d30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:30.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:30.004+0000 7f0fda000700 1 --2- 172.21.15.122:0/2561747781 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fd4103080 0x7f0fd41136c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:30.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:30.004+0000 7f0fda000700 1 -- 172.21.15.122:0/2561747781 >> 172.21.15.122:0/2561747781 conn(0x7f0fd4072300 msgr2=0x7f0fd4074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:30.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:30.004+0000 7f0fda000700 1 -- 172.21.15.122:0/2561747781 shutdown_connections 2024-04-16T05:33:30.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:30.004+0000 7f0fda000700 1 -- 172.21.15.122:0/2561747781 wait complete. 2024-04-16T05:33:30.004 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-16T05:33:31.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:30 smithi027 ceph-mon[29206]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:33:31.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:30 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/2561747781' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:33:31.598 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T05:33:31.599 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T05:33:32.057 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:32.157 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:33.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:32 smithi027 ceph-mon[29206]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:33:33.975 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.976+0000 7faca2800700 1 -- 172.21.15.122:0/1920301702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9c0fec70 msgr2=0x7fac9c0ff080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:33.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.976+0000 7faca2800700 1 --2- 172.21.15.122:0/1920301702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9c0fec70 0x7fac9c0ff080 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fac8c0048c0 tx=0x7fac8c009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:33.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.976+0000 7faca2800700 1 -- 172.21.15.122:0/1920301702 shutdown_connections 2024-04-16T05:33:33.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.976+0000 7faca2800700 1 --2- 172.21.15.122:0/1920301702 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9c0fec70 0x7fac9c0ff080 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:33.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.976+0000 7faca2800700 1 -- 172.21.15.122:0/1920301702 >> 172.21.15.122:0/1920301702 conn(0x7fac9c0fa5d0 msgr2=0x7fac9c0fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:33.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.976+0000 7faca2800700 1 -- 172.21.15.122:0/1920301702 shutdown_connections 2024-04-16T05:33:33.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.976+0000 7faca2800700 1 -- 172.21.15.122:0/1920301702 wait complete. 2024-04-16T05:33:33.976 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.977+0000 7faca2800700 1 Processor -- start 2024-04-16T05:33:33.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.978+0000 7faca2800700 1 -- start start 2024-04-16T05:33:33.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.978+0000 7faca2800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9c0fec70 0x7fac9c10f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:33.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.978+0000 7faca2800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c10f830 con 0x7fac9c0fec70 2024-04-16T05:33:33.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.978+0000 7fac9b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9c0fec70 0x7fac9c10f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:33.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.979+0000 7fac9b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9c0fec70 0x7fac9c10f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:46428/0 (socket says 172.21.15.122:46428) 2024-04-16T05:33:33.977 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.979+0000 7fac9b400700 1 -- 172.21.15.122:0/4261820289 learned_addr learned my addr 172.21.15.122:0/4261820289 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:33:33.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.979+0000 7fac9b400700 1 -- 172.21.15.122:0/4261820289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac8c005040 con 0x7fac9c0fec70 2024-04-16T05:33:33.978 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.979+0000 7fac9b400700 1 --2- 172.21.15.122:0/4261820289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9c0fec70 0x7fac9c10f2f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009680 tx=0x7fac8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:33.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.980+0000 7fac93400700 1 -- 172.21.15.122:0/4261820289 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac8c00b5c0 con 0x7fac9c0fec70 2024-04-16T05:33:33.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.980+0000 7faca2800700 1 -- 172.21.15.122:0/4261820289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac9c10fa30 con 0x7fac9c0fec70 2024-04-16T05:33:33.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.980+0000 7fac93400700 1 -- 172.21.15.122:0/4261820289 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fac8c00be70 con 0x7fac9c0fec70 2024-04-16T05:33:33.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.980+0000 7fac93400700 1 -- 172.21.15.122:0/4261820289 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac8c0135a0 con 0x7fac9c0fec70 2024-04-16T05:33:33.979 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.980+0000 7faca2800700 1 -- 172.21.15.122:0/4261820289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9c110550 con 0x7fac9c0fec70 2024-04-16T05:33:33.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.981+0000 7faca2800700 1 -- 172.21.15.122:0/4261820289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac9c105180 con 0x7fac9c0fec70 2024-04-16T05:33:33.980 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.982+0000 7fac93400700 1 -- 172.21.15.122:0/4261820289 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7fac8c004d30 con 0x7fac9c0fec70 2024-04-16T05:33:33.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.982+0000 7fac93400700 1 --2- 172.21.15.122:0/4261820289 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fac80036880 0x7fac80038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:33.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.982+0000 7fac93400700 1 -- 172.21.15.122:0/4261820289 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fac8c049460 con 0x7fac9c0fec70 2024-04-16T05:33:33.981 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.983+0000 7fac9aa00700 1 --2- 172.21.15.122:0/4261820289 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fac80036880 0x7fac80038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:33.982 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.984+0000 7fac9aa00700 1 --2- 172.21.15.122:0/4261820289 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fac80036880 0x7fac80038d30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fac88009990 tx=0x7fac88006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:33.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:33.997+0000 7fac93400700 1 -- 172.21.15.122:0/4261820289 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fac8c011960 con 0x7fac9c0fec70 2024-04-16T05:33:34.289 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:34.290+0000 7faca2800700 1 -- 172.21.15.122:0/4261820289 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac9c110f60 con 0x7fac9c0fec70 2024-04-16T05:33:34.291 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:34.291+0000 7fac93400700 1 -- 172.21.15.122:0/4261820289 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fac8c021070 con 0x7fac9c0fec70 2024-04-16T05:33:34.291 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:33:34.291 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","modified":"2024-04-16T05:30:56.860193Z","created":"2024-04-16T05:30:56.860193Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T05:33:34.293 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:34.294+0000 7faca2800700 1 -- 172.21.15.122:0/4261820289 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fac80036880 msgr2=0x7fac80038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:34.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:34.294+0000 7faca2800700 1 --2- 172.21.15.122:0/4261820289 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fac80036880 0x7fac80038d30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fac88009990 tx=0x7fac88006e40 comp rx=0 tx=0).stop 2024-04-16T05:33:34.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:34.294+0000 7faca2800700 1 -- 172.21.15.122:0/4261820289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9c0fec70 msgr2=0x7fac9c10f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:34.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:34.294+0000 7faca2800700 1 --2- 172.21.15.122:0/4261820289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9c0fec70 0x7fac9c10f2f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009680 tx=0x7fac8c009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:34.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:34.294+0000 7faca2800700 1 -- 172.21.15.122:0/4261820289 shutdown_connections 2024-04-16T05:33:34.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:34.294+0000 7faca2800700 1 --2- 172.21.15.122:0/4261820289 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7fac80036880 0x7fac80038d30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:34.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:34.294+0000 7faca2800700 1 --2- 172.21.15.122:0/4261820289 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fac9c0fec70 0x7fac9c10f2f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:34.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:34.294+0000 7faca2800700 1 -- 172.21.15.122:0/4261820289 >> 172.21.15.122:0/4261820289 conn(0x7fac9c0fa5d0 msgr2=0x7fac9c0fc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:34.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:34.295+0000 7faca2800700 1 -- 172.21.15.122:0/4261820289 shutdown_connections 2024-04-16T05:33:34.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:34.295+0000 7faca2800700 1 -- 172.21.15.122:0/4261820289 wait complete. 2024-04-16T05:33:34.295 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-16T05:33:34.714 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:34 smithi027 ceph-mon[29206]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:33:34.714 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:34 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/4261820289' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:33:35.897 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T05:33:35.898 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T05:33:36.327 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:36.427 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:36.898 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:36 smithi027 ceph-mon[29206]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:33:38.636 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.637+0000 7f06c8400700 1 -- 172.21.15.122:0/2524676098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06c0103080 msgr2=0x7f06c0103490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:38.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.637+0000 7f06c8400700 1 --2- 172.21.15.122:0/2524676098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06c0103080 0x7f06c0103490 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f06b40048c0 tx=0x7f06b4009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:38.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.637+0000 7f06c8400700 1 -- 172.21.15.122:0/2524676098 shutdown_connections 2024-04-16T05:33:38.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.637+0000 7f06c8400700 1 --2- 172.21.15.122:0/2524676098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06c0103080 0x7f06c0103490 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:38.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.637+0000 7f06c8400700 1 -- 172.21.15.122:0/2524676098 >> 172.21.15.122:0/2524676098 conn(0x7f06c0072300 msgr2=0x7f06c0074710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:38.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.637+0000 7f06c8400700 1 -- 172.21.15.122:0/2524676098 shutdown_connections 2024-04-16T05:33:38.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.637+0000 7f06c8400700 1 -- 172.21.15.122:0/2524676098 wait complete. 2024-04-16T05:33:38.637 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.638+0000 7f06c8400700 1 Processor -- start 2024-04-16T05:33:38.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.638+0000 7f06c8400700 1 -- start start 2024-04-16T05:33:38.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.639+0000 7f06c8400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06c0103080 0x7f06c01136c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:38.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.639+0000 7f06c8400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c0113c00 con 0x7f06c0103080 2024-04-16T05:33:38.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.639+0000 7f06c5a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06c0103080 0x7f06c01136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:38.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.639+0000 7f06c5a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06c0103080 0x7f06c01136c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:46436/0 (socket says 172.21.15.122:46436) 2024-04-16T05:33:38.638 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.639+0000 7f06c5a00700 1 -- 172.21.15.122:0/4104562946 learned_addr learned my addr 172.21.15.122:0/4104562946 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:33:38.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.639+0000 7f06c5a00700 1 -- 172.21.15.122:0/4104562946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b4005040 con 0x7f06c0103080 2024-04-16T05:33:38.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.640+0000 7f06c5a00700 1 --2- 172.21.15.122:0/4104562946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06c0103080 0x7f06c01136c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f06b4009680 tx=0x7f06b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:38.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.640+0000 7f06bd600700 1 -- 172.21.15.122:0/4104562946 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06b400b5c0 con 0x7f06c0103080 2024-04-16T05:33:38.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.640+0000 7f06bd600700 1 -- 172.21.15.122:0/4104562946 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f06b400be70 con 0x7f06c0103080 2024-04-16T05:33:38.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.640+0000 7f06c8400700 1 -- 172.21.15.122:0/4104562946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c0113e00 con 0x7f06c0103080 2024-04-16T05:33:38.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.640+0000 7f06bd600700 1 -- 172.21.15.122:0/4104562946 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06b40135a0 con 0x7f06c0103080 2024-04-16T05:33:38.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.640+0000 7f06c8400700 1 -- 172.21.15.122:0/4104562946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c0114970 con 0x7f06c0103080 2024-04-16T05:33:38.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.641+0000 7f06bd600700 1 -- 172.21.15.122:0/4104562946 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 13) v1 ==== 43432+0+0 (secure 0 0 0) 0x7f06b401c430 con 0x7f06c0103080 2024-04-16T05:33:38.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.641+0000 7f06c8400700 1 -- 172.21.15.122:0/4104562946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06c0109570 con 0x7f06c0103080 2024-04-16T05:33:38.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.642+0000 7f06bd600700 1 --2- 172.21.15.122:0/4104562946 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f06a80368d0 0x7f06a8038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:38.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.642+0000 7f06bd600700 1 -- 172.21.15.122:0/4104562946 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f06b4049320 con 0x7f06c0103080 2024-04-16T05:33:38.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.642+0000 7f06c5000700 1 --2- 172.21.15.122:0/4104562946 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f06a80368d0 0x7f06a8038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:38.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.643+0000 7f06c5000700 1 --2- 172.21.15.122:0/4104562946 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f06a80368d0 0x7f06a8038d80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f06ac009990 tx=0x7f06ac006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:38.648 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.650+0000 7f06bd600700 1 -- 172.21.15.122:0/4104562946 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f06b4011550 con 0x7f06c0103080 2024-04-16T05:33:38.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:38 smithi027 ceph-mon[29206]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:33:38.939 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.940+0000 7f06c8400700 1 -- 172.21.15.122:0/4104562946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06c0115380 con 0x7f06c0103080 2024-04-16T05:33:38.940 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.941+0000 7f06bd600700 1 -- 172.21.15.122:0/4104562946 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f06b4021070 con 0x7f06c0103080 2024-04-16T05:33:38.940 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:33:38.940 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","modified":"2024-04-16T05:30:56.860193Z","created":"2024-04-16T05:30:56.860193Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T05:33:38.943 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.944+0000 7f06c8400700 1 -- 172.21.15.122:0/4104562946 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f06a80368d0 msgr2=0x7f06a8038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:38.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.944+0000 7f06c8400700 1 --2- 172.21.15.122:0/4104562946 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f06a80368d0 0x7f06a8038d80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f06ac009990 tx=0x7f06ac006e40 comp rx=0 tx=0).stop 2024-04-16T05:33:38.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.944+0000 7f06c8400700 1 -- 172.21.15.122:0/4104562946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06c0103080 msgr2=0x7f06c01136c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:38.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.944+0000 7f06c8400700 1 --2- 172.21.15.122:0/4104562946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06c0103080 0x7f06c01136c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f06b4009680 tx=0x7f06b4009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:38.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.944+0000 7f06c8400700 1 -- 172.21.15.122:0/4104562946 shutdown_connections 2024-04-16T05:33:38.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.944+0000 7f06c8400700 1 --2- 172.21.15.122:0/4104562946 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f06a80368d0 0x7f06a8038d80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:38.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.944+0000 7f06c8400700 1 --2- 172.21.15.122:0/4104562946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06c0103080 0x7f06c01136c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:38.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.944+0000 7f06c8400700 1 -- 172.21.15.122:0/4104562946 >> 172.21.15.122:0/4104562946 conn(0x7f06c0072300 msgr2=0x7f06c0074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:38.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.944+0000 7f06c8400700 1 -- 172.21.15.122:0/4104562946 shutdown_connections 2024-04-16T05:33:38.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:38.945+0000 7f06c8400700 1 -- 172.21.15.122:0/4104562946 wait complete. 2024-04-16T05:33:38.944 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-16T05:33:40.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:39 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/4104562946' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:33:40.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:39 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:40.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:39 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:40.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:39 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:40.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:39 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-16T05:33:40.799 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T05:33:40.799 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T05:33:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:40 smithi027 ceph-mon[29206]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:33:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:40 smithi027 ceph-mon[29206]: from='mgr.14162 172.21.15.27:0/4237874770' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-16T05:33:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:40 smithi027 ceph-mon[29206]: mgrmap e14: smithi027.qgzypi(active, since 83s) 2024-04-16T05:33:41.249 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:41.344 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:43.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.300+0000 7eff40600700 1 -- 172.21.15.122:0/2935412078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff380669d0 msgr2=0x7eff380ff650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:43.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.300+0000 7eff40600700 1 --2- 172.21.15.122:0/2935412078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff380669d0 0x7eff380ff650 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7eff2c0048c0 tx=0x7eff2c009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:43.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.300+0000 7eff40600700 1 -- 172.21.15.122:0/2935412078 shutdown_connections 2024-04-16T05:33:43.299 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.300+0000 7eff40600700 1 --2- 172.21.15.122:0/2935412078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff380669d0 0x7eff380ff650 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:43.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.300+0000 7eff40600700 1 -- 172.21.15.122:0/2935412078 >> 172.21.15.122:0/2935412078 conn(0x7eff380f70c0 msgr2=0x7eff380f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:43.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.300+0000 7eff40600700 1 -- 172.21.15.122:0/2935412078 shutdown_connections 2024-04-16T05:33:43.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.300+0000 7eff40600700 1 -- 172.21.15.122:0/2935412078 wait complete. 2024-04-16T05:33:43.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.301+0000 7eff40600700 1 Processor -- start 2024-04-16T05:33:43.300 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.302+0000 7eff40600700 1 -- start start 2024-04-16T05:33:43.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.302+0000 7eff40600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff380669d0 0x7eff3810d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:43.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.302+0000 7eff40600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3810d630 con 0x7eff380669d0 2024-04-16T05:33:43.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.303+0000 7eff3dc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff380669d0 0x7eff3810d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:43.301 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.303+0000 7eff3dc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff380669d0 0x7eff3810d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:45006/0 (socket says 172.21.15.122:45006) 2024-04-16T05:33:43.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.303+0000 7eff3dc00700 1 -- 172.21.15.122:0/4139359406 learned_addr learned my addr 172.21.15.122:0/4139359406 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:33:43.302 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.304+0000 7eff3dc00700 1 -- 172.21.15.122:0/4139359406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2c005040 con 0x7eff380669d0 2024-04-16T05:33:43.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.304+0000 7eff3dc00700 1 --2- 172.21.15.122:0/4139359406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff380669d0 0x7eff3810d0f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7eff2c009680 tx=0x7eff2c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:43.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.305+0000 7eff35600700 1 -- 172.21.15.122:0/4139359406 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff2c00bed0 con 0x7eff380669d0 2024-04-16T05:33:43.303 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.305+0000 7eff40600700 1 -- 172.21.15.122:0/4139359406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff3810d830 con 0x7eff380669d0 2024-04-16T05:33:43.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.305+0000 7eff35600700 1 -- 172.21.15.122:0/4139359406 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff2c00b5c0 con 0x7eff380669d0 2024-04-16T05:33:43.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.305+0000 7eff35600700 1 -- 172.21.15.122:0/4139359406 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff2c0255c0 con 0x7eff380669d0 2024-04-16T05:33:43.304 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.305+0000 7eff40600700 1 -- 172.21.15.122:0/4139359406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff3810e3a0 con 0x7eff380669d0 2024-04-16T05:33:43.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.306+0000 7eff40600700 1 -- 172.21.15.122:0/4139359406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff38102f60 con 0x7eff380669d0 2024-04-16T05:33:43.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.306+0000 7eff35600700 1 -- 172.21.15.122:0/4139359406 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 43446+0+0 (secure 0 0 0) 0x7eff2c004d30 con 0x7eff380669d0 2024-04-16T05:33:43.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.307+0000 7eff35600700 1 --2- 172.21.15.122:0/4139359406 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7eff20036920 0x7eff20038dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:43.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.307+0000 7eff35600700 1 -- 172.21.15.122:0/4139359406 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7eff2c049a30 con 0x7eff380669d0 2024-04-16T05:33:43.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.307+0000 7eff3d200700 1 -- 172.21.15.122:0/4139359406 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7eff20036920 msgr2=0x7eff20038dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2700787222 2024-04-16T05:33:43.306 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.307+0000 7eff3d200700 1 --2- 172.21.15.122:0/4139359406 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7eff20036920 0x7eff20038dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T05:33:43.319 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.321+0000 7eff35600700 1 -- 172.21.15.122:0/4139359406 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7eff2c0238f0 con 0x7eff380669d0 2024-04-16T05:33:43.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.508+0000 7eff3d200700 1 -- 172.21.15.122:0/4139359406 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7eff20036920 msgr2=0x7eff20038dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2700787222 2024-04-16T05:33:43.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.508+0000 7eff3d200700 1 --2- 172.21.15.122:0/4139359406 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7eff20036920 0x7eff20038dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T05:33:43.609 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.610+0000 7eff40600700 1 -- 172.21.15.122:0/4139359406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff3805f5f0 con 0x7eff380669d0 2024-04-16T05:33:43.611 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.612+0000 7eff35600700 1 -- 172.21.15.122:0/4139359406 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7eff2c023300 con 0x7eff380669d0 2024-04-16T05:33:43.611 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:33:43.611 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","modified":"2024-04-16T05:30:56.860193Z","created":"2024-04-16T05:30:56.860193Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T05:33:43.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.614+0000 7eff40600700 1 -- 172.21.15.122:0/4139359406 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7eff20036920 msgr2=0x7eff20038dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T05:33:43.613 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.614+0000 7eff40600700 1 --2- 172.21.15.122:0/4139359406 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7eff20036920 0x7eff20038dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:43.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.614+0000 7eff40600700 1 -- 172.21.15.122:0/4139359406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff380669d0 msgr2=0x7eff3810d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:43.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.614+0000 7eff40600700 1 --2- 172.21.15.122:0/4139359406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff380669d0 0x7eff3810d0f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7eff2c009680 tx=0x7eff2c009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:43.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.615+0000 7eff40600700 1 -- 172.21.15.122:0/4139359406 shutdown_connections 2024-04-16T05:33:43.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.615+0000 7eff40600700 1 --2- 172.21.15.122:0/4139359406 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7eff20036920 0x7eff20038dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:43.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.615+0000 7eff40600700 1 --2- 172.21.15.122:0/4139359406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7eff380669d0 0x7eff3810d0f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:43.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.615+0000 7eff40600700 1 -- 172.21.15.122:0/4139359406 >> 172.21.15.122:0/4139359406 conn(0x7eff380f70c0 msgr2=0x7eff38103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:43.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.615+0000 7eff40600700 1 -- 172.21.15.122:0/4139359406 shutdown_connections 2024-04-16T05:33:43.614 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:43.615+0000 7eff40600700 1 -- 172.21.15.122:0/4139359406 wait complete. 2024-04-16T05:33:43.615 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-16T05:33:44.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:43 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/4139359406' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:33:45.289 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T05:33:45.290 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T05:33:45.737 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:45.841 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:47.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.741+0000 7f0ecf800700 1 -- 172.21.15.122:0/2561185036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ec8065cc0 msgr2=0x7f0ec80ff700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:47.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.741+0000 7f0ecf800700 1 --2- 172.21.15.122:0/2561185036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ec8065cc0 0x7f0ec80ff700 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc0048c0 tx=0x7f0ebc009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:47.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.742+0000 7f0ecf800700 1 -- 172.21.15.122:0/2561185036 shutdown_connections 2024-04-16T05:33:47.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.742+0000 7f0ecf800700 1 --2- 172.21.15.122:0/2561185036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ec8065cc0 0x7f0ec80ff700 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:47.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.742+0000 7f0ecf800700 1 -- 172.21.15.122:0/2561185036 >> 172.21.15.122:0/2561185036 conn(0x7f0ec80f70d0 msgr2=0x7f0ec80f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:47.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.742+0000 7f0ecf800700 1 -- 172.21.15.122:0/2561185036 shutdown_connections 2024-04-16T05:33:47.741 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.742+0000 7f0ecf800700 1 -- 172.21.15.122:0/2561185036 wait complete. 2024-04-16T05:33:47.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.743+0000 7f0ecf800700 1 Processor -- start 2024-04-16T05:33:47.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.743+0000 7f0ecf800700 1 -- start start 2024-04-16T05:33:47.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.744+0000 7f0ecf800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ec8065cc0 0x7f0ec810f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:47.742 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.744+0000 7f0ecf800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec810f820 con 0x7f0ec8065cc0 2024-04-16T05:33:47.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.744+0000 7f0ecce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ec8065cc0 0x7f0ec810f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:47.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.744+0000 7f0ecce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ec8065cc0 0x7f0ec810f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:45024/0 (socket says 172.21.15.122:45024) 2024-04-16T05:33:47.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.744+0000 7f0ecce00700 1 -- 172.21.15.122:0/2278612888 learned_addr learned my addr 172.21.15.122:0/2278612888 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:33:47.743 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.745+0000 7f0ecce00700 1 -- 172.21.15.122:0/2278612888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ebc005040 con 0x7f0ec8065cc0 2024-04-16T05:33:47.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.745+0000 7f0ecce00700 1 --2- 172.21.15.122:0/2278612888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ec8065cc0 0x7f0ec810f2e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009480 tx=0x7f0ebc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:47.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.746+0000 7f0ec4c00700 1 -- 172.21.15.122:0/2278612888 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ebc00bba0 con 0x7f0ec8065cc0 2024-04-16T05:33:47.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.746+0000 7f0ecf800700 1 -- 172.21.15.122:0/2278612888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec810fa20 con 0x7f0ec8065cc0 2024-04-16T05:33:47.744 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.746+0000 7f0ec4c00700 1 -- 172.21.15.122:0/2278612888 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ebc00bd00 con 0x7f0ec8065cc0 2024-04-16T05:33:47.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.746+0000 7f0ecf800700 1 -- 172.21.15.122:0/2278612888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec8110590 con 0x7f0ec8065cc0 2024-04-16T05:33:47.745 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.746+0000 7f0ec4c00700 1 -- 172.21.15.122:0/2278612888 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ebc01c550 con 0x7f0ec8065cc0 2024-04-16T05:33:47.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.747+0000 7f0ecf800700 1 -- 172.21.15.122:0/2278612888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec8102f80 con 0x7f0ec8065cc0 2024-04-16T05:33:47.746 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.747+0000 7f0ec4c00700 1 -- 172.21.15.122:0/2278612888 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 14) v1 ==== 43446+0+0 (secure 0 0 0) 0x7f0ebc013a80 con 0x7f0ec8065cc0 2024-04-16T05:33:47.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.748+0000 7f0ec4c00700 1 --2- 172.21.15.122:0/2278612888 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f0eb0036920 0x7f0eb0038dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:47.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.748+0000 7f0ec4c00700 1 -- 172.21.15.122:0/2278612888 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0ebc049530 con 0x7f0ec8065cc0 2024-04-16T05:33:47.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.748+0000 7f0ec7e00700 1 -- 172.21.15.122:0/2278612888 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f0eb0036920 msgr2=0x7f0eb0038dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2700787222 2024-04-16T05:33:47.747 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.748+0000 7f0ec7e00700 1 --2- 172.21.15.122:0/2278612888 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f0eb0036920 0x7f0eb0038dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T05:33:47.761 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.762+0000 7f0ec4c00700 1 -- 172.21.15.122:0/2278612888 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f0ebc013420 con 0x7f0ec8065cc0 2024-04-16T05:33:47.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.949+0000 7f0ec7e00700 1 -- 172.21.15.122:0/2278612888 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f0eb0036920 msgr2=0x7f0eb0038dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/2700787222 2024-04-16T05:33:47.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:47.949+0000 7f0ec7e00700 1 --2- 172.21.15.122:0/2278612888 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f0eb0036920 0x7f0eb0038dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T05:33:48.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:48.056+0000 7f0ecf800700 1 -- 172.21.15.122:0/2278612888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ec805f5f0 con 0x7f0ec8065cc0 2024-04-16T05:33:48.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:48.058+0000 7f0ec4c00700 1 -- 172.21.15.122:0/2278612888 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f0ebc021070 con 0x7f0ec8065cc0 2024-04-16T05:33:48.057 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:33:48.057 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","modified":"2024-04-16T05:30:56.860193Z","created":"2024-04-16T05:30:56.860193Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T05:33:48.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:48.060+0000 7f0ecf800700 1 -- 172.21.15.122:0/2278612888 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f0eb0036920 msgr2=0x7f0eb0038dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T05:33:48.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:48.061+0000 7f0ecf800700 1 --2- 172.21.15.122:0/2278612888 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f0eb0036920 0x7f0eb0038dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:48.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:48.061+0000 7f0ecf800700 1 -- 172.21.15.122:0/2278612888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ec8065cc0 msgr2=0x7f0ec810f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:48.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:48.061+0000 7f0ecf800700 1 --2- 172.21.15.122:0/2278612888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ec8065cc0 0x7f0ec810f2e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009480 tx=0x7f0ebc009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:48.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:48.061+0000 7f0ecf800700 1 -- 172.21.15.122:0/2278612888 shutdown_connections 2024-04-16T05:33:48.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:48.061+0000 7f0ecf800700 1 --2- 172.21.15.122:0/2278612888 >> [v2:172.21.15.27:6800/2700787222,v1:172.21.15.27:6801/2700787222] conn(0x7f0eb0036920 0x7f0eb0038dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:48.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:48.061+0000 7f0ecf800700 1 --2- 172.21.15.122:0/2278612888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ec8065cc0 0x7f0ec810f2e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:48.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:48.061+0000 7f0ecf800700 1 -- 172.21.15.122:0/2278612888 >> 172.21.15.122:0/2278612888 conn(0x7f0ec80f70d0 msgr2=0x7f0ec80f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:48.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:48.061+0000 7f0ecf800700 1 -- 172.21.15.122:0/2278612888 shutdown_connections 2024-04-16T05:33:48.061 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:48.061+0000 7f0ecf800700 1 -- 172.21.15.122:0/2278612888 wait complete. 2024-04-16T05:33:48.061 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-16T05:33:48.394 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:48 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/2278612888' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:33:49.722 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:49 smithi027 ceph-mon[29206]: Active manager daemon smithi027.qgzypi restarted 2024-04-16T05:33:49.722 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:49 smithi027 ceph-mon[29206]: Activating manager daemon smithi027.qgzypi 2024-04-16T05:33:49.722 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:49 smithi027 ceph-mon[29206]: osdmap e5: 0 total, 0 up, 0 in 2024-04-16T05:33:49.722 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:49 smithi027 ceph-mon[29206]: mgrmap e15: smithi027.qgzypi(active, starting, since 0.00938555s) 2024-04-16T05:33:49.722 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:49 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-16T05:33:49.722 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:49 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr metadata", "who": "smithi027.qgzypi", "id": "smithi027.qgzypi"}]: dispatch 2024-04-16T05:33:49.722 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:49 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T05:33:49.722 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:49 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T05:33:49.722 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:49 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T05:33:49.723 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:49 smithi027 ceph-mon[29206]: Manager daemon smithi027.qgzypi is now available 2024-04-16T05:33:49.793 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T05:33:49.793 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T05:33:50.231 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:50.326 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:50.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:50 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:33:50.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:50 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:33:50.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:50 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:50.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:50 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:33:50.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:50 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.qgzypi/mirror_snapshot_schedule"}]: dispatch 2024-04-16T05:33:50.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:50 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.qgzypi/trash_purge_schedule"}]: dispatch 2024-04-16T05:33:50.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:50 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:50.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:50 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:50.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:50 smithi027 ceph-mon[29206]: mgrmap e16: smithi027.qgzypi(active, since 1.01048s) 2024-04-16T05:33:51.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:51 smithi027 ceph-mon[29206]: [16/Apr/2024:05:33:49] ENGINE Bus STARTING 2024-04-16T05:33:51.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:51 smithi027 ceph-mon[29206]: [16/Apr/2024:05:33:50] ENGINE Serving on https://172.21.15.27:7150 2024-04-16T05:33:51.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:51 smithi027 ceph-mon[29206]: [16/Apr/2024:05:33:50] ENGINE Bus STARTED 2024-04-16T05:33:52.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.182+0000 7fd8fc000700 1 -- 172.21.15.122:0/965695990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8f40fca70 msgr2=0x7fd8f40fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:52.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.182+0000 7fd8fc000700 1 --2- 172.21.15.122:0/965695990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8f40fca70 0x7fd8f40fce80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd8e80048c0 tx=0x7fd8e8009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:52.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.182+0000 7fd8fc000700 1 -- 172.21.15.122:0/965695990 shutdown_connections 2024-04-16T05:33:52.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.182+0000 7fd8fc000700 1 --2- 172.21.15.122:0/965695990 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8f40fca70 0x7fd8f40fce80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:52.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.182+0000 7fd8fc000700 1 -- 172.21.15.122:0/965695990 >> 172.21.15.122:0/965695990 conn(0x7fd8f40f81d0 msgr2=0x7fd8f40fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:52.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.182+0000 7fd8fc000700 1 -- 172.21.15.122:0/965695990 shutdown_connections 2024-04-16T05:33:52.181 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.182+0000 7fd8fc000700 1 -- 172.21.15.122:0/965695990 wait complete. 2024-04-16T05:33:52.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.183+0000 7fd8fc000700 1 Processor -- start 2024-04-16T05:33:52.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.183+0000 7fd8fc000700 1 -- start start 2024-04-16T05:33:52.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.184+0000 7fd8fc000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8f40fca70 0x7fd8f410d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:52.182 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.184+0000 7fd8fc000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8f410d600 con 0x7fd8f40fca70 2024-04-16T05:33:52.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.184+0000 7fd8f9600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8f40fca70 0x7fd8f410d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:52.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.184+0000 7fd8f9600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8f40fca70 0x7fd8f410d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:49640/0 (socket says 172.21.15.122:49640) 2024-04-16T05:33:52.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.184+0000 7fd8f9600700 1 -- 172.21.15.122:0/1859621384 learned_addr learned my addr 172.21.15.122:0/1859621384 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:33:52.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.185+0000 7fd8f9600700 1 -- 172.21.15.122:0/1859621384 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e8005040 con 0x7fd8f40fca70 2024-04-16T05:33:52.183 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.185+0000 7fd8f9600700 1 --2- 172.21.15.122:0/1859621384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8f40fca70 0x7fd8f410d0c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8009680 tx=0x7fd8e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:52.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.185+0000 7fd8f1600700 1 -- 172.21.15.122:0/1859621384 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8e800b5c0 con 0x7fd8f40fca70 2024-04-16T05:33:52.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.185+0000 7fd8fc000700 1 -- 172.21.15.122:0/1859621384 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8f410d800 con 0x7fd8f40fca70 2024-04-16T05:33:52.184 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.185+0000 7fd8f1600700 1 -- 172.21.15.122:0/1859621384 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd8e800be70 con 0x7fd8f40fca70 2024-04-16T05:33:52.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.185+0000 7fd8f1600700 1 -- 172.21.15.122:0/1859621384 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8e80135a0 con 0x7fd8f40fca70 2024-04-16T05:33:52.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.185+0000 7fd8fc000700 1 -- 172.21.15.122:0/1859621384 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8f410e370 con 0x7fd8f40fca70 2024-04-16T05:33:52.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.186+0000 7fd8f1600700 1 -- 172.21.15.122:0/1859621384 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7fd8e801ca90 con 0x7fd8f40fca70 2024-04-16T05:33:52.185 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.186+0000 7fd8fc000700 1 -- 172.21.15.122:0/1859621384 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8f4102f80 con 0x7fd8f40fca70 2024-04-16T05:33:52.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.187+0000 7fd8f1600700 1 --2- 172.21.15.122:0/1859621384 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd8dc0369c0 0x7fd8dc038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:52.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.187+0000 7fd8f1600700 1 -- 172.21.15.122:0/1859621384 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fd8e8049570 con 0x7fd8f40fca70 2024-04-16T05:33:52.186 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.187+0000 7fd8f8c00700 1 --2- 172.21.15.122:0/1859621384 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd8dc0369c0 0x7fd8dc038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:52.187 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.189+0000 7fd8f8c00700 1 --2- 172.21.15.122:0/1859621384 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd8dc0369c0 0x7fd8dc038e70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0009990 tx=0x7fd8e0006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:52.193 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.195+0000 7fd8f1600700 1 -- 172.21.15.122:0/1859621384 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fd8e8011960 con 0x7fd8f40fca70 2024-04-16T05:33:52.491 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.492+0000 7fd8fc000700 1 -- 172.21.15.122:0/1859621384 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd8f410ee40 con 0x7fd8f40fca70 2024-04-16T05:33:52.492 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.494+0000 7fd8f1600700 1 -- 172.21.15.122:0/1859621384 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fd8e8021070 con 0x7fd8f40fca70 2024-04-16T05:33:52.493 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:33:52.493 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","modified":"2024-04-16T05:30:56.860193Z","created":"2024-04-16T05:30:56.860193Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T05:33:52.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.497+0000 7fd8fc000700 1 -- 172.21.15.122:0/1859621384 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd8dc0369c0 msgr2=0x7fd8dc038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:52.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.497+0000 7fd8fc000700 1 --2- 172.21.15.122:0/1859621384 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd8dc0369c0 0x7fd8dc038e70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0009990 tx=0x7fd8e0006e40 comp rx=0 tx=0).stop 2024-04-16T05:33:52.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.497+0000 7fd8fc000700 1 -- 172.21.15.122:0/1859621384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8f40fca70 msgr2=0x7fd8f410d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:52.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.497+0000 7fd8fc000700 1 --2- 172.21.15.122:0/1859621384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8f40fca70 0x7fd8f410d0c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8009680 tx=0x7fd8e8009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:52.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.497+0000 7fd8fc000700 1 -- 172.21.15.122:0/1859621384 shutdown_connections 2024-04-16T05:33:52.496 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.497+0000 7fd8fc000700 1 --2- 172.21.15.122:0/1859621384 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd8dc0369c0 0x7fd8dc038e70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:52.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.497+0000 7fd8fc000700 1 --2- 172.21.15.122:0/1859621384 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8f40fca70 0x7fd8f410d0c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:52.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.497+0000 7fd8fc000700 1 -- 172.21.15.122:0/1859621384 >> 172.21.15.122:0/1859621384 conn(0x7fd8f40f81d0 msgr2=0x7fd8f4103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:52.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.497+0000 7fd8fc000700 1 -- 172.21.15.122:0/1859621384 shutdown_connections 2024-04-16T05:33:52.497 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:52.497+0000 7fd8fc000700 1 -- 172.21.15.122:0/1859621384 wait complete. 2024-04-16T05:33:52.498 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-16T05:33:52.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:52 smithi027 ceph-mon[29206]: mgrmap e17: smithi027.qgzypi(active, since 2s) 2024-04-16T05:33:52.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:52 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:52.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:52 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/1859621384' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:33:54.142 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T05:33:54.142 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T05:33:54.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:54 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:54.827 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:55.350 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /etc/ceph/ceph.conf 2024-04-16T05:33:56.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:56 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:56.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:56 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:57.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.628+0000 7f0114600700 1 -- 172.21.15.122:0/345163359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f010c071ea0 msgr2=0x7f010c0703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:57.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.628+0000 7f0114600700 1 --2- 172.21.15.122:0/345163359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f010c071ea0 0x7f010c0703b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f01000048c0 tx=0x7f0100009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:57.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.628+0000 7f0114600700 1 -- 172.21.15.122:0/345163359 shutdown_connections 2024-04-16T05:33:57.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.628+0000 7f0114600700 1 --2- 172.21.15.122:0/345163359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f010c071ea0 0x7f010c0703b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:57.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.629+0000 7f0114600700 1 -- 172.21.15.122:0/345163359 >> 172.21.15.122:0/345163359 conn(0x7f010c0f81e0 msgr2=0x7f010c0fa610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:57.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.629+0000 7f0114600700 1 -- 172.21.15.122:0/345163359 shutdown_connections 2024-04-16T05:33:57.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.629+0000 7f0114600700 1 -- 172.21.15.122:0/345163359 wait complete. 2024-04-16T05:33:57.898 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.630+0000 7f0114600700 1 Processor -- start 2024-04-16T05:33:57.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.630+0000 7f0114600700 1 -- start start 2024-04-16T05:33:57.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.630+0000 7f0114600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f010c071ea0 0x7f010c10f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:57.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.630+0000 7f0114600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010c10f830 con 0x7f010c071ea0 2024-04-16T05:33:57.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.631+0000 7f0111c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f010c071ea0 0x7f010c10f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:57.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.631+0000 7f0111c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f010c071ea0 0x7f010c10f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:49660/0 (socket says 172.21.15.122:49660) 2024-04-16T05:33:57.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.631+0000 7f0111c00700 1 -- 172.21.15.122:0/1216546504 learned_addr learned my addr 172.21.15.122:0/1216546504 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:33:57.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.632+0000 7f0111c00700 1 -- 172.21.15.122:0/1216546504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0100005040 con 0x7f010c071ea0 2024-04-16T05:33:57.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.632+0000 7f0111c00700 1 --2- 172.21.15.122:0/1216546504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f010c071ea0 0x7f010c10f2f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f010000b900 tx=0x7f0100009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:57.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.633+0000 7f0109600700 1 -- 172.21.15.122:0/1216546504 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f010000b5c0 con 0x7f010c071ea0 2024-04-16T05:33:57.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.633+0000 7f0114600700 1 -- 172.21.15.122:0/1216546504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f010c10fa30 con 0x7f010c071ea0 2024-04-16T05:33:57.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.633+0000 7f0109600700 1 -- 172.21.15.122:0/1216546504 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f010000be70 con 0x7f010c071ea0 2024-04-16T05:33:57.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.633+0000 7f0109600700 1 -- 172.21.15.122:0/1216546504 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f010001c590 con 0x7f010c071ea0 2024-04-16T05:33:57.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.633+0000 7f0114600700 1 -- 172.21.15.122:0/1216546504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f010c1105a0 con 0x7f010c071ea0 2024-04-16T05:33:57.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.634+0000 7f0114600700 1 -- 172.21.15.122:0/1216546504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f010c1051a0 con 0x7f010c071ea0 2024-04-16T05:33:57.899 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.635+0000 7f0109600700 1 -- 172.21.15.122:0/1216546504 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7f0100004d30 con 0x7f010c071ea0 2024-04-16T05:33:57.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.635+0000 7f0109600700 1 --2- 172.21.15.122:0/1216546504 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f00f4036970 0x7f00f4038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:33:57.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.635+0000 7f0109600700 1 -- 172.21.15.122:0/1216546504 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f01000495a0 con 0x7f010c071ea0 2024-04-16T05:33:57.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.636+0000 7f0111200700 1 --2- 172.21.15.122:0/1216546504 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f00f4036970 0x7f00f4038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:33:57.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.637+0000 7f0111200700 1 --2- 172.21.15.122:0/1216546504 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f00f4036970 0x7f00f4038e20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f00f8009990 tx=0x7f00f8006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:33:57.900 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.650+0000 7f0109600700 1 -- 172.21.15.122:0/1216546504 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f010001b350 con 0x7f010c071ea0 2024-04-16T05:33:57.944 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.946+0000 7f0114600700 1 -- 172.21.15.122:0/1216546504 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f010c05f640 con 0x7f010c071ea0 2024-04-16T05:33:57.945 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.946+0000 7f0109600700 1 -- 172.21.15.122:0/1216546504 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f0100021030 con 0x7f010c071ea0 2024-04-16T05:33:57.945 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:33:57.945 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","modified":"2024-04-16T05:30:56.860193Z","created":"2024-04-16T05:30:56.860193Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T05:33:57.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.949+0000 7f0114600700 1 -- 172.21.15.122:0/1216546504 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f00f4036970 msgr2=0x7f00f4038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:57.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.949+0000 7f0114600700 1 --2- 172.21.15.122:0/1216546504 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f00f4036970 0x7f00f4038e20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f00f8009990 tx=0x7f00f8006e40 comp rx=0 tx=0).stop 2024-04-16T05:33:57.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.949+0000 7f0114600700 1 -- 172.21.15.122:0/1216546504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f010c071ea0 msgr2=0x7f010c10f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:33:57.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.949+0000 7f0114600700 1 --2- 172.21.15.122:0/1216546504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f010c071ea0 0x7f010c10f2f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f010000b900 tx=0x7f0100009e30 comp rx=0 tx=0).stop 2024-04-16T05:33:57.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.949+0000 7f0114600700 1 -- 172.21.15.122:0/1216546504 shutdown_connections 2024-04-16T05:33:57.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.949+0000 7f0114600700 1 --2- 172.21.15.122:0/1216546504 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f00f4036970 0x7f00f4038e20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:57.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.949+0000 7f0114600700 1 --2- 172.21.15.122:0/1216546504 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f010c071ea0 0x7f010c10f2f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:33:57.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.949+0000 7f0114600700 1 -- 172.21.15.122:0/1216546504 >> 172.21.15.122:0/1216546504 conn(0x7f010c0f81e0 msgr2=0x7f010c105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:33:57.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.949+0000 7f0114600700 1 -- 172.21.15.122:0/1216546504 shutdown_connections 2024-04-16T05:33:57.948 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:33:57.949+0000 7f0114600700 1 -- 172.21.15.122:0/1216546504 wait complete. 2024-04-16T05:33:57.949 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-16T05:33:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:58 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:58 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:58 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd/host:smithi027", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:33:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:58 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:58 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:58 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:58 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:33:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:58 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:33:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:58 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:33:58.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:58 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:33:58.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:58 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/1216546504' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:33:59.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:59 smithi027 ceph-mon[29206]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-16T05:33:59.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:59 smithi027 ceph-mon[29206]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-16T05:33:59.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:59 smithi027 ceph-mon[29206]: Updating smithi122:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:33:59.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:33:59 smithi027 ceph-mon[29206]: Updating smithi027:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:33:59.730 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T05:33:59.730 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T05:34:00.149 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:34:00.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:00 smithi027 ceph-mon[29206]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-16T05:34:00.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:00 smithi027 ceph-mon[29206]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-16T05:34:02.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:01 smithi027 ceph-mon[29206]: Updating smithi122:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T05:34:02.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:01 smithi027 ceph-mon[29206]: Updating smithi027:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T05:34:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:01 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:01 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:01 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:01 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:01 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:01 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T05:34:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:01 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-16T05:34:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:01 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:34:02.264 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.258+0000 7f42abe00700 1 -- 172.21.15.122:0/2807854631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac0702e0 msgr2=0x7f42ac0706b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:02.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.258+0000 7f42abe00700 1 --2- 172.21.15.122:0/2807854631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac0702e0 0x7f42ac0706b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f429c0048c0 tx=0x7f429c009e30 comp rx=0 tx=0).stop 2024-04-16T05:34:02.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.258+0000 7f42abe00700 1 -- 172.21.15.122:0/2807854631 shutdown_connections 2024-04-16T05:34:02.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.258+0000 7f42abe00700 1 --2- 172.21.15.122:0/2807854631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac0702e0 0x7f42ac0706b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:02.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.258+0000 7f42abe00700 1 -- 172.21.15.122:0/2807854631 >> 172.21.15.122:0/2807854631 conn(0x7f42ac0f8450 msgr2=0x7f42ac0fa860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:34:02.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.258+0000 7f42abe00700 1 -- 172.21.15.122:0/2807854631 shutdown_connections 2024-04-16T05:34:02.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.258+0000 7f42abe00700 1 -- 172.21.15.122:0/2807854631 wait complete. 2024-04-16T05:34:02.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.259+0000 7f42abe00700 1 Processor -- start 2024-04-16T05:34:02.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.259+0000 7f42abe00700 1 -- start start 2024-04-16T05:34:02.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.260+0000 7f42abe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac0702e0 0x7f42ac115210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:34:02.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.260+0000 7f42abe00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac115750 con 0x7f42ac0702e0 2024-04-16T05:34:02.265 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.261+0000 7f42aaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac0702e0 0x7f42ac115210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:34:02.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.261+0000 7f42aaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac0702e0 0x7f42ac115210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:34574/0 (socket says 172.21.15.122:34574) 2024-04-16T05:34:02.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.261+0000 7f42aaa00700 1 -- 172.21.15.122:0/3864378153 learned_addr learned my addr 172.21.15.122:0/3864378153 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:34:02.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.261+0000 7f42aaa00700 1 -- 172.21.15.122:0/3864378153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429c005040 con 0x7f42ac0702e0 2024-04-16T05:34:02.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.262+0000 7f42aaa00700 1 --2- 172.21.15.122:0/3864378153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac0702e0 0x7f42ac115210 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f429c00b7f0 tx=0x7f429c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:34:02.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.262+0000 7f42a2a00700 1 -- 172.21.15.122:0/3864378153 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f429c009370 con 0x7f42ac0702e0 2024-04-16T05:34:02.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.263+0000 7f42a2a00700 1 -- 172.21.15.122:0/3864378153 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f429c013420 con 0x7f42ac0702e0 2024-04-16T05:34:02.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.263+0000 7f42a2a00700 1 -- 172.21.15.122:0/3864378153 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f429c01c590 con 0x7f42ac0702e0 2024-04-16T05:34:02.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.263+0000 7f42abe00700 1 -- 172.21.15.122:0/3864378153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42ac115950 con 0x7f42ac0702e0 2024-04-16T05:34:02.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.263+0000 7f42abe00700 1 -- 172.21.15.122:0/3864378153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42ac115dc0 con 0x7f42ac0702e0 2024-04-16T05:34:02.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.264+0000 7f42abe00700 1 -- 172.21.15.122:0/3864378153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42ac071a20 con 0x7f42ac0702e0 2024-04-16T05:34:02.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.264+0000 7f42a2a00700 1 -- 172.21.15.122:0/3864378153 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7f429c013a70 con 0x7f42ac0702e0 2024-04-16T05:34:02.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.265+0000 7f42a2a00700 1 --2- 172.21.15.122:0/3864378153 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f42900369c0 0x7f4290038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:34:02.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.265+0000 7f42a2a00700 1 -- 172.21.15.122:0/3864378153 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f429c0495a0 con 0x7f42ac0702e0 2024-04-16T05:34:02.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.266+0000 7f42aa000700 1 --2- 172.21.15.122:0/3864378153 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f42900369c0 0x7f4290038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:34:02.267 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.267+0000 7f42aa000700 1 --2- 172.21.15.122:0/3864378153 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f42900369c0 0x7f4290038e70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4298009990 tx=0x7f4298006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:34:02.280 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.281+0000 7f42a2a00700 1 -- 172.21.15.122:0/3864378153 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f429c024b50 con 0x7f42ac0702e0 2024-04-16T05:34:02.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.588+0000 7f42abe00700 1 -- 172.21.15.122:0/3864378153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42ac05f5f0 con 0x7f42ac0702e0 2024-04-16T05:34:02.588 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.589+0000 7f42a2a00700 1 -- 172.21.15.122:0/3864378153 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f429c021030 con 0x7f42ac0702e0 2024-04-16T05:34:02.589 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:34:02.589 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","modified":"2024-04-16T05:30:56.860193Z","created":"2024-04-16T05:30:56.860193Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T05:34:02.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.592+0000 7f42abe00700 1 -- 172.21.15.122:0/3864378153 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f42900369c0 msgr2=0x7f4290038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:02.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.592+0000 7f42abe00700 1 --2- 172.21.15.122:0/3864378153 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f42900369c0 0x7f4290038e70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4298009990 tx=0x7f4298006e40 comp rx=0 tx=0).stop 2024-04-16T05:34:02.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.592+0000 7f42abe00700 1 -- 172.21.15.122:0/3864378153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac0702e0 msgr2=0x7f42ac115210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:02.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.592+0000 7f42abe00700 1 --2- 172.21.15.122:0/3864378153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac0702e0 0x7f42ac115210 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f429c00b7f0 tx=0x7f429c009e30 comp rx=0 tx=0).stop 2024-04-16T05:34:02.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.593+0000 7f42abe00700 1 -- 172.21.15.122:0/3864378153 shutdown_connections 2024-04-16T05:34:02.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.593+0000 7f42abe00700 1 --2- 172.21.15.122:0/3864378153 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f42900369c0 0x7f4290038e70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:02.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.593+0000 7f42abe00700 1 --2- 172.21.15.122:0/3864378153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42ac0702e0 0x7f42ac115210 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:02.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.593+0000 7f42abe00700 1 -- 172.21.15.122:0/3864378153 >> 172.21.15.122:0/3864378153 conn(0x7f42ac0f8450 msgr2=0x7f42ac105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:34:02.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.593+0000 7f42abe00700 1 -- 172.21.15.122:0/3864378153 shutdown_connections 2024-04-16T05:34:02.592 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:02.593+0000 7f42abe00700 1 -- 172.21.15.122:0/3864378153 wait complete. 2024-04-16T05:34:02.593 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-16T05:34:03.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:02 smithi027 ceph-mon[29206]: Deploying daemon crash.smithi122 on smithi122 2024-04-16T05:34:03.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:02 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/3864378153' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:34:04.235 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T05:34:04.235 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T05:34:04.724 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:34:05.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:04 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:34:07.004 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.004+0000 7ffbbd400700 1 -- 172.21.15.122:0/2266435459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbb8102b00 msgr2=0x7ffbb8102ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:07.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.004+0000 7ffbbd400700 1 --2- 172.21.15.122:0/2266435459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbb8102b00 0x7ffbb8102ed0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ffba80048c0 tx=0x7ffba8009e30 comp rx=0 tx=0).stop 2024-04-16T05:34:07.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.004+0000 7ffbbd400700 1 -- 172.21.15.122:0/2266435459 shutdown_connections 2024-04-16T05:34:07.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.004+0000 7ffbbd400700 1 --2- 172.21.15.122:0/2266435459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbb8102b00 0x7ffbb8102ed0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:07.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.004+0000 7ffbbd400700 1 -- 172.21.15.122:0/2266435459 >> 172.21.15.122:0/2266435459 conn(0x7ffbb80f8600 msgr2=0x7ffbb80faa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:34:07.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.004+0000 7ffbbd400700 1 -- 172.21.15.122:0/2266435459 shutdown_connections 2024-04-16T05:34:07.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.004+0000 7ffbbd400700 1 -- 172.21.15.122:0/2266435459 wait complete. 2024-04-16T05:34:07.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.005+0000 7ffbbd400700 1 Processor -- start 2024-04-16T05:34:07.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.006+0000 7ffbbd400700 1 -- start start 2024-04-16T05:34:07.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.006+0000 7ffbbd400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbb8102b00 0x7ffbb81153f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:34:07.005 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.006+0000 7ffbbd400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb8115930 con 0x7ffbb8102b00 2024-04-16T05:34:07.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.006+0000 7ffbb6800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbb8102b00 0x7ffbb81153f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:34:07.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.007+0000 7ffbb6800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbb8102b00 0x7ffbb81153f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:34608/0 (socket says 172.21.15.122:34608) 2024-04-16T05:34:07.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.007+0000 7ffbb6800700 1 -- 172.21.15.122:0/3015007146 learned_addr learned my addr 172.21.15.122:0/3015007146 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:34:07.006 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.007+0000 7ffbb6800700 1 -- 172.21.15.122:0/3015007146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba8005040 con 0x7ffbb8102b00 2024-04-16T05:34:07.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.008+0000 7ffbb6800700 1 --2- 172.21.15.122:0/3015007146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbb8102b00 0x7ffbb81153f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffba800b7f0 tx=0x7ffba8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:34:07.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.008+0000 7ffbaea00700 1 -- 172.21.15.122:0/3015007146 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffba8009370 con 0x7ffbb8102b00 2024-04-16T05:34:07.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.008+0000 7ffbbd400700 1 -- 172.21.15.122:0/3015007146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb8115b30 con 0x7ffbb8102b00 2024-04-16T05:34:07.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.008+0000 7ffbbd400700 1 -- 172.21.15.122:0/3015007146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb8116020 con 0x7ffbb8102b00 2024-04-16T05:34:07.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.008+0000 7ffbaea00700 1 -- 172.21.15.122:0/3015007146 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffba8013420 con 0x7ffbb8102b00 2024-04-16T05:34:07.007 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.008+0000 7ffbaea00700 1 -- 172.21.15.122:0/3015007146 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffba801c570 con 0x7ffbb8102b00 2024-04-16T05:34:07.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.009+0000 7ffbbd400700 1 -- 172.21.15.122:0/3015007146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb940052b0 con 0x7ffbb8102b00 2024-04-16T05:34:07.008 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.010+0000 7ffbaea00700 1 -- 172.21.15.122:0/3015007146 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7ffba8013a50 con 0x7ffbb8102b00 2024-04-16T05:34:07.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.010+0000 7ffbaea00700 1 --2- 172.21.15.122:0/3015007146 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7ffb9c036970 0x7ffb9c038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:34:07.009 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.010+0000 7ffbaea00700 1 -- 172.21.15.122:0/3015007146 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7ffba80499f0 con 0x7ffbb8102b00 2024-04-16T05:34:07.010 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.011+0000 7ffbafe00700 1 --2- 172.21.15.122:0/3015007146 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7ffb9c036970 0x7ffb9c038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:34:07.011 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.013+0000 7ffbafe00700 1 --2- 172.21.15.122:0/3015007146 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7ffb9c036970 0x7ffb9c038e20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffba0009990 tx=0x7ffba0006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:34:07.024 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.026+0000 7ffbaea00700 1 -- 172.21.15.122:0/3015007146 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ffba801b350 con 0x7ffbb8102b00 2024-04-16T05:34:07.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.327+0000 7ffbbd400700 1 -- 172.21.15.122:0/3015007146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb94005560 con 0x7ffbb8102b00 2024-04-16T05:34:07.326 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.328+0000 7ffbaea00700 1 -- 172.21.15.122:0/3015007146 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7ffba802e300 con 0x7ffbb8102b00 2024-04-16T05:34:07.327 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:34:07.327 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","modified":"2024-04-16T05:30:56.860193Z","created":"2024-04-16T05:30:56.860193Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T05:34:07.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.330+0000 7ffbbd400700 1 -- 172.21.15.122:0/3015007146 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7ffb9c036970 msgr2=0x7ffb9c038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:07.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.330+0000 7ffbbd400700 1 --2- 172.21.15.122:0/3015007146 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7ffb9c036970 0x7ffb9c038e20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffba0009990 tx=0x7ffba0006e40 comp rx=0 tx=0).stop 2024-04-16T05:34:07.329 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.330+0000 7ffbbd400700 1 -- 172.21.15.122:0/3015007146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbb8102b00 msgr2=0x7ffbb81153f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:07.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.330+0000 7ffbbd400700 1 --2- 172.21.15.122:0/3015007146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbb8102b00 0x7ffbb81153f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ffba800b7f0 tx=0x7ffba8009e30 comp rx=0 tx=0).stop 2024-04-16T05:34:07.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.330+0000 7ffbbd400700 1 -- 172.21.15.122:0/3015007146 shutdown_connections 2024-04-16T05:34:07.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.330+0000 7ffbbd400700 1 --2- 172.21.15.122:0/3015007146 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7ffb9c036970 0x7ffb9c038e20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:07.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.330+0000 7ffbbd400700 1 --2- 172.21.15.122:0/3015007146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffbb8102b00 0x7ffbb81153f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:07.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.330+0000 7ffbbd400700 1 -- 172.21.15.122:0/3015007146 >> 172.21.15.122:0/3015007146 conn(0x7ffbb80f8600 msgr2=0x7ffbb80f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:34:07.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.330+0000 7ffbbd400700 1 -- 172.21.15.122:0/3015007146 shutdown_connections 2024-04-16T05:34:07.330 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:07.331+0000 7ffbbd400700 1 -- 172.21.15.122:0/3015007146 wait complete. 2024-04-16T05:34:07.330 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-16T05:34:08.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:08 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:08.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:08 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:08.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:08 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:08.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:08 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:08.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:08 smithi027 ceph-mon[29206]: Deploying daemon node-exporter.smithi122 on smithi122 2024-04-16T05:34:08.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:08 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/3015007146' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:34:09.056 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T05:34:09.057 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T05:34:09.514 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:34:10.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:10 smithi027 ceph-mon[29206]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:10.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:10 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:12.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:12 smithi027 ceph-mon[29206]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:13.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.895+0000 7f7cea200700 1 -- 172.21.15.122:0/2630952259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce4104c90 msgr2=0x7f7ce4105060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:13.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.895+0000 7f7cea200700 1 --2- 172.21.15.122:0/2630952259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce4104c90 0x7f7ce4105060 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40048c0 tx=0x7f7cd4009e30 comp rx=0 tx=0).stop 2024-04-16T05:34:13.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.895+0000 7f7cea200700 1 -- 172.21.15.122:0/2630952259 shutdown_connections 2024-04-16T05:34:13.103 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.895+0000 7f7cea200700 1 --2- 172.21.15.122:0/2630952259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce4104c90 0x7f7ce4105060 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:13.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.895+0000 7f7cea200700 1 -- 172.21.15.122:0/2630952259 >> 172.21.15.122:0/2630952259 conn(0x7f7ce40fab10 msgr2=0x7f7ce40fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:34:13.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.895+0000 7f7cea200700 1 -- 172.21.15.122:0/2630952259 shutdown_connections 2024-04-16T05:34:13.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.895+0000 7f7cea200700 1 -- 172.21.15.122:0/2630952259 wait complete. 2024-04-16T05:34:13.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.896+0000 7f7cea200700 1 Processor -- start 2024-04-16T05:34:13.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.896+0000 7f7cea200700 1 -- start start 2024-04-16T05:34:13.104 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.896+0000 7f7cea200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce4104c90 0x7f7ce4115330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:34:13.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.896+0000 7f7cea200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce4115870 con 0x7f7ce4104c90 2024-04-16T05:34:13.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.897+0000 7f7ce3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce4104c90 0x7f7ce4115330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:34:13.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.897+0000 7f7ce3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce4104c90 0x7f7ce4115330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:53792/0 (socket says 172.21.15.122:53792) 2024-04-16T05:34:13.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.897+0000 7f7ce3400700 1 -- 172.21.15.122:0/3834100744 learned_addr learned my addr 172.21.15.122:0/3834100744 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:34:13.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.897+0000 7f7ce3400700 1 -- 172.21.15.122:0/3834100744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd4005040 con 0x7f7ce4104c90 2024-04-16T05:34:13.105 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.898+0000 7f7ce3400700 1 --2- 172.21.15.122:0/3834100744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce4104c90 0x7f7ce4115330 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7cd400b7f0 tx=0x7f7cd4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:34:13.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.898+0000 7f7cdb400700 1 -- 172.21.15.122:0/3834100744 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cd4009370 con 0x7f7ce4104c90 2024-04-16T05:34:13.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.898+0000 7f7cea200700 1 -- 172.21.15.122:0/3834100744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce4115a70 con 0x7f7ce4104c90 2024-04-16T05:34:13.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.898+0000 7f7cea200700 1 -- 172.21.15.122:0/3834100744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce4115f60 con 0x7f7ce4104c90 2024-04-16T05:34:13.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.898+0000 7f7cdb400700 1 -- 172.21.15.122:0/3834100744 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7cd4013420 con 0x7f7ce4104c90 2024-04-16T05:34:13.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.898+0000 7f7cdb400700 1 -- 172.21.15.122:0/3834100744 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7cd401c570 con 0x7f7ce4104c90 2024-04-16T05:34:13.106 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.900+0000 7f7cea200700 1 -- 172.21.15.122:0/3834100744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce41003e0 con 0x7f7ce4104c90 2024-04-16T05:34:13.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.900+0000 7f7cdb400700 1 -- 172.21.15.122:0/3834100744 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7f7cd4013a50 con 0x7f7ce4104c90 2024-04-16T05:34:13.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.900+0000 7f7cdb400700 1 --2- 172.21.15.122:0/3834100744 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f7cc80369c0 0x7f7cc8038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:34:13.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.900+0000 7f7cdb400700 1 -- 172.21.15.122:0/3834100744 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f7cd40493f0 con 0x7f7ce4104c90 2024-04-16T05:34:13.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.901+0000 7f7ce2a00700 1 --2- 172.21.15.122:0/3834100744 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f7cc80369c0 0x7f7cc8038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:34:13.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.902+0000 7f7ce2a00700 1 --2- 172.21.15.122:0/3834100744 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f7cc80369c0 0x7f7cc8038e70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009990 tx=0x7f7cd0006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:34:13.107 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:12.915+0000 7f7cdb400700 1 -- 172.21.15.122:0/3834100744 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f7cd4024770 con 0x7f7ce4104c90 2024-04-16T05:34:13.246 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:13.247+0000 7f7cea200700 1 -- 172.21.15.122:0/3834100744 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ce405f5f0 con 0x7f7ce4104c90 2024-04-16T05:34:13.247 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:13.248+0000 7f7cdb400700 1 -- 172.21.15.122:0/3834100744 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f7cd4021030 con 0x7f7ce4104c90 2024-04-16T05:34:13.247 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:34:13.248 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","modified":"2024-04-16T05:30:56.860193Z","created":"2024-04-16T05:30:56.860193Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T05:34:13.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:13.251+0000 7f7cea200700 1 -- 172.21.15.122:0/3834100744 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f7cc80369c0 msgr2=0x7f7cc8038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:13.250 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:13.251+0000 7f7cea200700 1 --2- 172.21.15.122:0/3834100744 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f7cc80369c0 0x7f7cc8038e70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009990 tx=0x7f7cd0006e40 comp rx=0 tx=0).stop 2024-04-16T05:34:13.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:13.251+0000 7f7cea200700 1 -- 172.21.15.122:0/3834100744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce4104c90 msgr2=0x7f7ce4115330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:13.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:13.251+0000 7f7cea200700 1 --2- 172.21.15.122:0/3834100744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce4104c90 0x7f7ce4115330 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7cd400b7f0 tx=0x7f7cd4009e30 comp rx=0 tx=0).stop 2024-04-16T05:34:13.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:13.251+0000 7f7cea200700 1 -- 172.21.15.122:0/3834100744 shutdown_connections 2024-04-16T05:34:13.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:13.251+0000 7f7cea200700 1 --2- 172.21.15.122:0/3834100744 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f7cc80369c0 0x7f7cc8038e70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:13.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:13.251+0000 7f7cea200700 1 --2- 172.21.15.122:0/3834100744 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ce4104c90 0x7f7ce4115330 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:13.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:13.251+0000 7f7cea200700 1 -- 172.21.15.122:0/3834100744 >> 172.21.15.122:0/3834100744 conn(0x7f7ce40fab10 msgr2=0x7f7ce40fb3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:34:13.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:13.251+0000 7f7cea200700 1 -- 172.21.15.122:0/3834100744 shutdown_connections 2024-04-16T05:34:13.251 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:13.252+0000 7f7cea200700 1 -- 172.21.15.122:0/3834100744 wait complete. 2024-04-16T05:34:13.252 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-16T05:34:14.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:14 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:14.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:14 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:14.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:14 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:14.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:14 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:14.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:14 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.sgokkq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T05:34:14.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:14 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.sgokkq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-16T05:34:14.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:14 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T05:34:14.559 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:14 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:34:14.559 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:14 smithi027 ceph-mon[29206]: Deploying daemon mgr.smithi122.sgokkq on smithi122 2024-04-16T05:34:14.559 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:14 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/3834100744' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:34:14.559 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:14 smithi027 ceph-mon[29206]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:15.042 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T05:34:15.042 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T05:34:15.535 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:34:16.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:15 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:17.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:16 smithi027 ceph-mon[29206]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:17.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.826+0000 7fc913e00700 1 -- 172.21.15.122:0/1773037898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc90c070360 msgr2=0x7fc90c070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:17.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.826+0000 7fc913e00700 1 --2- 172.21.15.122:0/1773037898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc90c070360 0x7fc90c070730 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc9000048c0 tx=0x7fc900009e30 comp rx=0 tx=0).stop 2024-04-16T05:34:17.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.827+0000 7fc913e00700 1 -- 172.21.15.122:0/1773037898 shutdown_connections 2024-04-16T05:34:17.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.827+0000 7fc913e00700 1 --2- 172.21.15.122:0/1773037898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc90c070360 0x7fc90c070730 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:17.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.827+0000 7fc913e00700 1 -- 172.21.15.122:0/1773037898 >> 172.21.15.122:0/1773037898 conn(0x7fc90c0f85f0 msgr2=0x7fc90c0faa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:34:17.826 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.827+0000 7fc913e00700 1 -- 172.21.15.122:0/1773037898 shutdown_connections 2024-04-16T05:34:17.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.827+0000 7fc913e00700 1 -- 172.21.15.122:0/1773037898 wait complete. 2024-04-16T05:34:17.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.828+0000 7fc913e00700 1 Processor -- start 2024-04-16T05:34:17.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.828+0000 7fc913e00700 1 -- start start 2024-04-16T05:34:17.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.828+0000 7fc913e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc90c070360 0x7fc90c1153a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:34:17.827 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.828+0000 7fc913e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90c1158e0 con 0x7fc90c070360 2024-04-16T05:34:17.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.829+0000 7fc911400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc90c070360 0x7fc90c1153a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:34:17.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.829+0000 7fc911400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc90c070360 0x7fc90c1153a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:53816/0 (socket says 172.21.15.122:53816) 2024-04-16T05:34:17.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.829+0000 7fc911400700 1 -- 172.21.15.122:0/3605635787 learned_addr learned my addr 172.21.15.122:0/3605635787 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:34:17.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.829+0000 7fc911400700 1 -- 172.21.15.122:0/3605635787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc900005040 con 0x7fc90c070360 2024-04-16T05:34:17.828 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.830+0000 7fc911400700 1 --2- 172.21.15.122:0/3605635787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc90c070360 0x7fc90c1153a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc90000b7f0 tx=0x7fc900009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:34:17.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.830+0000 7fc909600700 1 -- 172.21.15.122:0/3605635787 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc900009370 con 0x7fc90c070360 2024-04-16T05:34:17.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.830+0000 7fc913e00700 1 -- 172.21.15.122:0/3605635787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc90c115ae0 con 0x7fc90c070360 2024-04-16T05:34:17.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.830+0000 7fc913e00700 1 -- 172.21.15.122:0/3605635787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc90c115f50 con 0x7fc90c070360 2024-04-16T05:34:17.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.830+0000 7fc909600700 1 -- 172.21.15.122:0/3605635787 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc900013420 con 0x7fc90c070360 2024-04-16T05:34:17.829 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.830+0000 7fc909600700 1 -- 172.21.15.122:0/3605635787 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc90001c570 con 0x7fc90c070360 2024-04-16T05:34:17.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.831+0000 7fc913e00700 1 -- 172.21.15.122:0/3605635787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc90c071aa0 con 0x7fc90c070360 2024-04-16T05:34:17.830 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.832+0000 7fc909600700 1 -- 172.21.15.122:0/3605635787 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7fc900013a50 con 0x7fc90c070360 2024-04-16T05:34:17.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.833+0000 7fc909600700 1 --2- 172.21.15.122:0/3605635787 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc8f40369c0 0x7fc8f4038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:34:17.831 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.833+0000 7fc909600700 1 -- 172.21.15.122:0/3605635787 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fc900049430 con 0x7fc90c070360 2024-04-16T05:34:17.832 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.833+0000 7fc910a00700 1 --2- 172.21.15.122:0/3605635787 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc8f40369c0 0x7fc8f4038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:34:17.833 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.834+0000 7fc910a00700 1 --2- 172.21.15.122:0/3605635787 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc8f40369c0 0x7fc8f4038e70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8009990 tx=0x7fc8f8006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:34:17.846 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:17.847+0000 7fc909600700 1 -- 172.21.15.122:0/3605635787 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fc900024770 con 0x7fc90c070360 2024-04-16T05:34:18.144 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:18.145+0000 7fc913e00700 1 -- 172.21.15.122:0/3605635787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc90c04bed0 con 0x7fc90c070360 2024-04-16T05:34:18.145 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:18.146+0000 7fc909600700 1 -- 172.21.15.122:0/3605635787 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fc900021030 con 0x7fc90c070360 2024-04-16T05:34:18.145 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:34:18.145 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","modified":"2024-04-16T05:30:56.860193Z","created":"2024-04-16T05:30:56.860193Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T05:34:18.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:18.149+0000 7fc913e00700 1 -- 172.21.15.122:0/3605635787 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc8f40369c0 msgr2=0x7fc8f4038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:18.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:18.149+0000 7fc913e00700 1 --2- 172.21.15.122:0/3605635787 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc8f40369c0 0x7fc8f4038e70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8009990 tx=0x7fc8f8006e40 comp rx=0 tx=0).stop 2024-04-16T05:34:18.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:18.149+0000 7fc913e00700 1 -- 172.21.15.122:0/3605635787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc90c070360 msgr2=0x7fc90c1153a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:18.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:18.149+0000 7fc913e00700 1 --2- 172.21.15.122:0/3605635787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc90c070360 0x7fc90c1153a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc90000b7f0 tx=0x7fc900009e30 comp rx=0 tx=0).stop 2024-04-16T05:34:18.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:18.149+0000 7fc913e00700 1 -- 172.21.15.122:0/3605635787 shutdown_connections 2024-04-16T05:34:18.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:18.149+0000 7fc913e00700 1 --2- 172.21.15.122:0/3605635787 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc8f40369c0 0x7fc8f4038e70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:18.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:18.149+0000 7fc913e00700 1 --2- 172.21.15.122:0/3605635787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc90c070360 0x7fc90c1153a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:18.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:18.149+0000 7fc913e00700 1 -- 172.21.15.122:0/3605635787 >> 172.21.15.122:0/3605635787 conn(0x7fc90c0f85f0 msgr2=0x7fc90c105950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:34:18.148 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:18.149+0000 7fc913e00700 1 -- 172.21.15.122:0/3605635787 shutdown_connections 2024-04-16T05:34:18.149 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:18.149+0000 7fc913e00700 1 -- 172.21.15.122:0/3605635787 wait complete. 2024-04-16T05:34:18.149 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-16T05:34:19.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:18 smithi027 ceph-mon[29206]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:19.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:18 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:19.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:18 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:19.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:18 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:19.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:18 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:19.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:18 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T05:34:19.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:18 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:34:19.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:18 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/3605635787' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:34:19.914 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T05:34:19.914 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T05:34:20.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:19 smithi027 ceph-mon[29206]: Deploying daemon mon.smithi122 on smithi122 2024-04-16T05:34:20.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:19 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:34:20.504 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:34:20.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:20 smithi027 ceph-mon[29206]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:20.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:20 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:23.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.780+0000 7f7a4ae00700 1 -- 172.21.15.122:0/3350069598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a441028f0 msgr2=0x7f7a44102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:23.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.780+0000 7f7a4ae00700 1 --2- 172.21.15.122:0/3350069598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a441028f0 0x7f7a44102cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7a380048c0 tx=0x7f7a38009e30 comp rx=0 tx=0).stop 2024-04-16T05:34:23.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.780+0000 7f7a4ae00700 1 -- 172.21.15.122:0/3350069598 shutdown_connections 2024-04-16T05:34:23.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.780+0000 7f7a4ae00700 1 --2- 172.21.15.122:0/3350069598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a441028f0 0x7f7a44102cc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:23.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.780+0000 7f7a4ae00700 1 -- 172.21.15.122:0/3350069598 >> 172.21.15.122:0/3350069598 conn(0x7f7a440f83f0 msgr2=0x7f7a440fa800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:34:23.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.780+0000 7f7a4ae00700 1 -- 172.21.15.122:0/3350069598 shutdown_connections 2024-04-16T05:34:23.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.780+0000 7f7a4ae00700 1 -- 172.21.15.122:0/3350069598 wait complete. 2024-04-16T05:34:23.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.782+0000 7f7a4ae00700 1 Processor -- start 2024-04-16T05:34:23.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.782+0000 7f7a4ae00700 1 -- start start 2024-04-16T05:34:23.055 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.783+0000 7f7a4ae00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a441028f0 0x7f7a44112fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:34:23.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.783+0000 7f7a4ae00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a441134e0 con 0x7f7a441028f0 2024-04-16T05:34:23.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.783+0000 7f7a49a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a441028f0 0x7f7a44112fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:34:23.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.783+0000 7f7a49a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a441028f0 0x7f7a44112fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:52482/0 (socket says 172.21.15.122:52482) 2024-04-16T05:34:23.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.783+0000 7f7a49a00700 1 -- 172.21.15.122:0/2993808960 learned_addr learned my addr 172.21.15.122:0/2993808960 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:34:23.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.784+0000 7f7a49a00700 1 -- 172.21.15.122:0/2993808960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a38005040 con 0x7f7a441028f0 2024-04-16T05:34:23.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.784+0000 7f7a49a00700 1 --2- 172.21.15.122:0/2993808960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a441028f0 0x7f7a44112fa0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800b7f0 tx=0x7f7a38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:34:23.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.785+0000 7f7a41600700 1 -- 172.21.15.122:0/2993808960 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a38009370 con 0x7f7a441028f0 2024-04-16T05:34:23.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.785+0000 7f7a4ae00700 1 -- 172.21.15.122:0/2993808960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a441136e0 con 0x7f7a441028f0 2024-04-16T05:34:23.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.785+0000 7f7a41600700 1 -- 172.21.15.122:0/2993808960 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a38013420 con 0x7f7a441028f0 2024-04-16T05:34:23.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.785+0000 7f7a41600700 1 -- 172.21.15.122:0/2993808960 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a3801c570 con 0x7f7a441028f0 2024-04-16T05:34:23.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.785+0000 7f7a4ae00700 1 -- 172.21.15.122:0/2993808960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a44113bd0 con 0x7f7a441028f0 2024-04-16T05:34:23.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.786+0000 7f7a4ae00700 1 -- 172.21.15.122:0/2993808960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a440fdff0 con 0x7f7a441028f0 2024-04-16T05:34:23.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.786+0000 7f7a41600700 1 -- 172.21.15.122:0/2993808960 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 17) v1 ==== 43489+0+0 (secure 0 0 0) 0x7f7a38013a50 con 0x7f7a441028f0 2024-04-16T05:34:23.056 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.787+0000 7f7a41600700 1 --2- 172.21.15.122:0/2993808960 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f7a2c0369c0 0x7f7a2c038e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:34:23.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.787+0000 7f7a41600700 1 -- 172.21.15.122:0/2993808960 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f7a38049430 con 0x7f7a441028f0 2024-04-16T05:34:23.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.788+0000 7f7a49000700 1 --2- 172.21.15.122:0/2993808960 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f7a2c0369c0 0x7f7a2c038e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:34:23.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.789+0000 7f7a49000700 1 --2- 172.21.15.122:0/2993808960 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f7a2c0369c0 0x7f7a2c038e70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7a30009990 tx=0x7f7a30006e40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:34:23.057 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:22.802+0000 7f7a41600700 1 -- 172.21.15.122:0/2993808960 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f7a38024770 con 0x7f7a441028f0 2024-04-16T05:34:23.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:22 smithi027 ceph-mon[29206]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:23.127 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:23.127+0000 7f7a4ae00700 1 -- 172.21.15.122:0/2993808960 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a4405f5f0 con 0x7f7a441028f0 2024-04-16T05:34:23.128 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:23.129+0000 7f7a41600700 1 -- 172.21.15.122:0/2993808960 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f7a38021030 con 0x7f7a441028f0 2024-04-16T05:34:23.128 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:34:23.128 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":1,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","modified":"2024-04-16T05:30:56.860193Z","created":"2024-04-16T05:30:56.860193Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-16T05:34:23.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:23.133+0000 7f7a4ae00700 1 -- 172.21.15.122:0/2993808960 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f7a2c0369c0 msgr2=0x7f7a2c038e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:23.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:23.133+0000 7f7a4ae00700 1 --2- 172.21.15.122:0/2993808960 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f7a2c0369c0 0x7f7a2c038e70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7a30009990 tx=0x7f7a30006e40 comp rx=0 tx=0).stop 2024-04-16T05:34:23.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:23.133+0000 7f7a4ae00700 1 -- 172.21.15.122:0/2993808960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a441028f0 msgr2=0x7f7a44112fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:23.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:23.133+0000 7f7a4ae00700 1 --2- 172.21.15.122:0/2993808960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a441028f0 0x7f7a44112fa0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800b7f0 tx=0x7f7a38009e30 comp rx=0 tx=0).stop 2024-04-16T05:34:23.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:23.133+0000 7f7a4ae00700 1 -- 172.21.15.122:0/2993808960 shutdown_connections 2024-04-16T05:34:23.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:23.133+0000 7f7a4ae00700 1 --2- 172.21.15.122:0/2993808960 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f7a2c0369c0 0x7f7a2c038e70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:23.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:23.133+0000 7f7a4ae00700 1 --2- 172.21.15.122:0/2993808960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a441028f0 0x7f7a44112fa0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:23.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:23.133+0000 7f7a4ae00700 1 -- 172.21.15.122:0/2993808960 >> 172.21.15.122:0/2993808960 conn(0x7f7a440f83f0 msgr2=0x7f7a440f9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:34:23.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:23.133+0000 7f7a4ae00700 1 -- 172.21.15.122:0/2993808960 shutdown_connections 2024-04-16T05:34:23.132 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:23.133+0000 7f7a4ae00700 1 -- 172.21.15.122:0/2993808960 wait complete. 2024-04-16T05:34:23.133 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 1 2024-04-16T05:34:23.630 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:23 smithi122 systemd[1]: Starting Ceph mon.smithi122 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T05:34:24.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:23 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/2993808960' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:34:24.606 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 podman[35976]: 2024-04-16 05:34:24.202423482 +0000 UTC m=+0.033323048 image pull b333581ccdf5c328f64509c4ba16a1a468c02f5eb459fd8c7d1e1ce535883bbe quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77 2024-04-16T05:34:24.606 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 podman[35976]: 2024-04-16 05:34:24.337509195 +0000 UTC m=+0.168408800 container create 5df19eab6d214c734d17d4f0a3c4f2c1cdf7e7b05541c0171eb1bdf91fd855bd (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi122, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T05:34:24.607 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 podman[35976]: 2024-04-16 05:34:24.60770547 +0000 UTC m=+0.438605095 container init 5df19eab6d214c734d17d4f0a3c4f2c1cdf7e7b05541c0171eb1bdf91fd855bd (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi122, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-64502-quincy-3-45317f9, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-16T05:34:25.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 podman[35976]: 2024-04-16 05:34:24.6209597 +0000 UTC m=+0.451859275 container start 5df19eab6d214c734d17d4f0a3c4f2c1cdf7e7b05541c0171eb1bdf91fd855bd (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi122, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-64502-quincy-3-45317f9, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-16T05:34:25.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: set uid:gid to 167:167 (ceph:ceph) 2024-04-16T05:34:25.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable), process ceph-mon, pid 2 2024-04-16T05:34:25.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: pidfile_write: ignore empty --pid-file 2024-04-16T05:34:25.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: load: jerasure load: lrc 2024-04-16T05:34:25.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: RocksDB version: 6.15.5 2024-04-16T05:34:25.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-16T05:34:25.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Compile date Apr 12 2024 2024-04-16T05:34:25.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: DB SUMMARY 2024-04-16T05:34:25.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: DB Session ID: QE4YAT23XEBKZOEDZ89X 2024-04-16T05:34:25.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: CURRENT file: CURRENT 2024-04-16T05:34:25.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: IDENTITY file: IDENTITY 2024-04-16T05:34:25.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-16T05:34:25.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi122/store.db dir, Total Num: 0, files: 2024-04-16T05:34:25.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi122/store.db: 000004.log size: 511 ; 2024-04-16T05:34:25.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.error_if_exists: 0 2024-04-16T05:34:25.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.create_if_missing: 0 2024-04-16T05:34:25.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.paranoid_checks: 1 2024-04-16T05:34:25.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-16T05:34:25.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.env: 0x55d011143b00 2024-04-16T05:34:25.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.fs: Posix File System 2024-04-16T05:34:25.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.info_log: 0x55d0480adc00 2024-04-16T05:34:25.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_file_opening_threads: 16 2024-04-16T05:34:25.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.statistics: (nil) 2024-04-16T05:34:25.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.use_fsync: 0 2024-04-16T05:34:25.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_log_file_size: 0 2024-04-16T05:34:25.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-16T05:34:25.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-16T05:34:25.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.keep_log_file_num: 1000 2024-04-16T05:34:25.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.recycle_log_file_num: 0 2024-04-16T05:34:25.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.allow_fallocate: 1 2024-04-16T05:34:25.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.allow_mmap_reads: 0 2024-04-16T05:34:25.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.allow_mmap_writes: 0 2024-04-16T05:34:25.028 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.use_direct_reads: 0 2024-04-16T05:34:25.028 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-16T05:34:25.028 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.create_missing_column_families: 0 2024-04-16T05:34:25.029 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.db_log_dir: 2024-04-16T05:34:25.029 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi122/store.db 2024-04-16T05:34:25.029 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-16T05:34:25.029 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-16T05:34:25.029 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-16T05:34:25.029 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-16T05:34:25.029 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-16T05:34:25.030 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-16T05:34:25.030 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.advise_random_on_open: 1 2024-04-16T05:34:25.030 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.db_write_buffer_size: 0 2024-04-16T05:34:25.030 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.write_buffer_manager: 0x55d048e22a20 2024-04-16T05:34:25.030 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-16T05:34:25.030 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-16T05:34:25.030 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-16T05:34:25.031 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-16T05:34:25.031 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.rate_limiter: (nil) 2024-04-16T05:34:25.031 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-16T05:34:25.031 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.wal_recovery_mode: 2 2024-04-16T05:34:25.031 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.enable_thread_tracking: 0 2024-04-16T05:34:25.031 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.enable_pipelined_write: 0 2024-04-16T05:34:25.031 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.unordered_write: 0 2024-04-16T05:34:25.031 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-16T05:34:25.031 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-16T05:34:25.032 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-16T05:34:25.032 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-16T05:34:25.032 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.row_cache: None 2024-04-16T05:34:25.032 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.wal_filter: None 2024-04-16T05:34:25.032 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-16T05:34:25.032 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.allow_ingest_behind: 0 2024-04-16T05:34:25.032 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.preserve_deletes: 0 2024-04-16T05:34:25.032 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.two_write_queues: 0 2024-04-16T05:34:25.033 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.manual_wal_flush: 0 2024-04-16T05:34:25.033 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.atomic_flush: 0 2024-04-16T05:34:25.033 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-16T05:34:25.033 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-16T05:34:25.033 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-16T05:34:25.033 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.log_readahead_size: 0 2024-04-16T05:34:25.033 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-16T05:34:25.033 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.best_efforts_recovery: 0 2024-04-16T05:34:25.034 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-16T05:34:25.034 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-16T05:34:25.034 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.allow_data_in_errors: 0 2024-04-16T05:34:25.034 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.db_host_id: __hostname__ 2024-04-16T05:34:25.034 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_background_jobs: 2 2024-04-16T05:34:25.034 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_background_compactions: -1 2024-04-16T05:34:25.034 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_subcompactions: 1 2024-04-16T05:34:25.034 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-16T05:34:25.035 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-16T05:34:25.035 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-16T05:34:25.035 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_total_wal_size: 0 2024-04-16T05:34:25.035 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-16T05:34:25.035 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-16T05:34:25.035 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-16T05:34:25.035 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-16T05:34:25.035 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_open_files: -1 2024-04-16T05:34:25.036 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.bytes_per_sync: 0 2024-04-16T05:34:25.036 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-16T05:34:25.036 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-16T05:34:25.036 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.compaction_readahead_size: 0 2024-04-16T05:34:25.036 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_background_flushes: -1 2024-04-16T05:34:25.036 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Compression algorithms supported: 2024-04-16T05:34:25.036 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-16T05:34:25.036 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: kZSTD supported: 0 2024-04-16T05:34:25.036 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: kXpressCompression supported: 0 2024-04-16T05:34:25.037 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: kLZ4HCCompression supported: 1 2024-04-16T05:34:25.037 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: kZlibCompression supported: 1 2024-04-16T05:34:25.037 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: kSnappyCompression supported: 1 2024-04-16T05:34:25.037 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: kLZ4Compression supported: 1 2024-04-16T05:34:25.037 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: kBZip2Compression supported: 0 2024-04-16T05:34:25.037 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-16T05:34:25.037 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi122/store.db/MANIFEST-000003 2024-04-16T05:34:25.037 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-04-16T05:34:25.038 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-16T05:34:25.038 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.merge_operator: 2024-04-16T05:34:25.038 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.compaction_filter: None 2024-04-16T05:34:25.038 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.compaction_filter_factory: None 2024-04-16T05:34:25.038 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.sst_partitioner_factory: None 2024-04-16T05:34:25.038 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-16T05:34:25.038 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-16T05:34:25.038 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d048102f58) 2024-04-16T05:34:25.038 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: cache_index_and_filter_blocks: 1 2024-04-16T05:34:25.038 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-16T05:34:25.038 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-16T05:34:25.038 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: pin_top_level_index_and_filter: 1 2024-04-16T05:34:25.039 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: index_type: 0 2024-04-16T05:34:25.039 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: data_block_index_type: 0 2024-04-16T05:34:25.039 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: index_shortening: 1 2024-04-16T05:34:25.039 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-16T05:34:25.039 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: hash_index_allow_collision: 1 2024-04-16T05:34:25.039 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: checksum: 1 2024-04-16T05:34:25.039 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: no_block_cache: 0 2024-04-16T05:34:25.039 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_cache: 0x55d048135090 2024-04-16T05:34:25.040 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_cache_name: BinnedLRUCache 2024-04-16T05:34:25.040 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_cache_options: 2024-04-16T05:34:25.040 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: capacity : 536870912 2024-04-16T05:34:25.040 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: num_shard_bits : 4 2024-04-16T05:34:25.040 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: strict_capacity_limit : 0 2024-04-16T05:34:25.040 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: high_pri_pool_ratio: 0.000 2024-04-16T05:34:25.040 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_cache_compressed: (nil) 2024-04-16T05:34:25.041 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: persistent_cache: (nil) 2024-04-16T05:34:25.041 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_size: 4096 2024-04-16T05:34:25.041 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_size_deviation: 10 2024-04-16T05:34:25.041 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_restart_interval: 16 2024-04-16T05:34:25.041 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: index_block_restart_interval: 1 2024-04-16T05:34:25.041 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: metadata_block_size: 4096 2024-04-16T05:34:25.041 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: partition_filters: 0 2024-04-16T05:34:25.041 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: use_delta_encoding: 1 2024-04-16T05:34:25.041 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-16T05:34:25.041 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: whole_key_filtering: 1 2024-04-16T05:34:25.042 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: verify_compression: 0 2024-04-16T05:34:25.042 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: read_amp_bytes_per_bit: 0 2024-04-16T05:34:25.042 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: format_version: 4 2024-04-16T05:34:25.042 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: enable_index_compression: 1 2024-04-16T05:34:25.042 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_align: 0 2024-04-16T05:34:25.042 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.write_buffer_size: 33554432 2024-04-16T05:34:25.042 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_write_buffer_number: 2 2024-04-16T05:34:25.042 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.compression: NoCompression 2024-04-16T05:34:25.042 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.bottommost_compression: Disabled 2024-04-16T05:34:25.042 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.prefix_extractor: nullptr 2024-04-16T05:34:25.042 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-16T05:34:25.042 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.num_levels: 7 2024-04-16T05:34:25.042 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-16T05:34:25.043 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-16T05:34:25.043 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-16T05:34:25.043 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-16T05:34:25.043 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-16T05:34:25.043 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-16T05:34:25.043 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-16T05:34:25.043 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-16T05:34:25.043 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-16T05:34:25.043 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-16T05:34:25.043 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-16T05:34:25.043 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.compression_opts.level: 32767 2024-04-16T05:34:25.043 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.compression_opts.strategy: 0 2024-04-16T05:34:25.043 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-16T05:34:25.043 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-16T05:34:25.043 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-16T05:34:25.044 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.compression_opts.enabled: false 2024-04-16T05:34:25.044 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-16T05:34:25.044 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-16T05:34:25.044 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-16T05:34:25.044 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.target_file_size_base: 67108864 2024-04-16T05:34:25.044 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-16T05:34:25.044 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-16T05:34:25.044 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-16T05:34:25.044 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-16T05:34:25.044 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-16T05:34:25.044 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-16T05:34:25.044 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-16T05:34:25.044 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-16T05:34:25.044 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-16T05:34:25.044 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-16T05:34:25.044 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-16T05:34:25.044 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-16T05:34:25.045 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-16T05:34:25.045 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.arena_block_size: 4194304 2024-04-16T05:34:25.045 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-16T05:34:25.045 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-16T05:34:25.045 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-16T05:34:25.045 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.disable_auto_compactions: 0 2024-04-16T05:34:25.045 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-16T05:34:25.045 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-16T05:34:25.045 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-16T05:34:25.045 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-16T05:34:25.045 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-16T05:34:25.045 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-16T05:34:25.045 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-16T05:34:25.045 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-16T05:34:25.045 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-16T05:34:25.045 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-16T05:34:25.046 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.table_properties_collectors: 2024-04-16T05:34:25.046 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.inplace_update_support: 0 2024-04-16T05:34:25.046 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-16T05:34:25.046 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-16T05:34:25.046 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-16T05:34:25.046 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-16T05:34:25.046 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.bloom_locality: 0 2024-04-16T05:34:25.046 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.max_successive_merges: 0 2024-04-16T05:34:25.046 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-16T05:34:25.046 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.paranoid_file_checks: 0 2024-04-16T05:34:25.046 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.force_consistency_checks: 1 2024-04-16T05:34:25.046 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.report_bg_io_stats: 0 2024-04-16T05:34:25.046 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.ttl: 2592000 2024-04-16T05:34:25.046 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-16T05:34:25.046 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.enable_blob_files: false 2024-04-16T05:34:25.047 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.min_blob_size: 0 2024-04-16T05:34:25.047 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.blob_file_size: 268435456 2024-04-16T05:34:25.047 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-16T05:34:25.047 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-16T05:34:25.047 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-16T05:34:25.047 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi122/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-16T05:34:25.047 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-04-16T05:34:25.047 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-04-16T05:34:25.047 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713245664672716, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-16T05:34:25.047 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-04-16T05:34:25.047 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-16T05:34:25.047 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713245664673793, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713245664, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "ec05ef60-e035-4e5f-b4a8-0e053e1ad4c8", "db_session_id": "QE4YAT23XEBKZOEDZ89X"}} 2024-04-16T05:34:25.047 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-04-16T05:34:25.047 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713245664674768, "job": 1, "event": "recovery_finished"} 2024-04-16T05:34:25.047 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi122/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-16T05:34:25.047 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55d0480c9180 2024-04-16T05:34:25.048 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: DB pointer 0x55d04819e000 2024-04-16T05:34:25.048 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-04-16T05:34:25.048 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-16T05:34:25.048 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ** DB Stats ** 2024-04-16T05:34:25.048 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T05:34:25.048 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-16T05:34:25.048 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T05:34:25.048 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T05:34:25.048 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-16T05:34:25.048 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-16T05:34:25.048 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T05:34:25.048 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-16T05:34:25.048 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ** Compaction Stats [default] ** 2024-04-16T05:34:25.048 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-16T05:34:25.048 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T05:34:25.048 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 2024-04-16T05:34:25.049 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 2024-04-16T05:34:25.049 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 2024-04-16T05:34:25.049 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-16T05:34:25.049 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ** Compaction Stats [default] ** 2024-04-16T05:34:25.049 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-16T05:34:25.049 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T05:34:25.049 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.4 0.00 0.00 1 0.001 0 0 2024-04-16T05:34:25.049 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T05:34:25.049 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-16T05:34:25.049 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-16T05:34:25.049 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-16T05:34:25.049 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-16T05:34:25.049 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-16T05:34:25.049 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Cumulative compaction: 0.00 GB write, 0.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T05:34:25.049 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Interval compaction: 0.00 GB write, 0.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T05:34:25.050 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-16T05:34:25.050 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-16T05:34:25.050 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-16T05:34:25.050 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-16T05:34:25.050 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ** Compaction Stats [default] ** 2024-04-16T05:34:25.050 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-16T05:34:25.050 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T05:34:25.050 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 2024-04-16T05:34:25.050 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 2024-04-16T05:34:25.050 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-16T05:34:25.050 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-16T05:34:25.050 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ** Compaction Stats [default] ** 2024-04-16T05:34:25.050 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-16T05:34:25.050 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T05:34:25.051 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.4 0.00 0.00 1 0.001 0 0 2024-04-16T05:34:25.051 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T05:34:25.051 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-16T05:34:25.051 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-16T05:34:25.051 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-16T05:34:25.051 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-16T05:34:25.051 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-16T05:34:25.051 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Cumulative compaction: 0.00 GB write, 0.15 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T05:34:25.051 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T05:34:25.051 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-16T05:34:25.051 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-16T05:34:25.051 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-16T05:34:25.051 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: mon.smithi122 does not exist in monmap, will attempt to join an existing cluster 2024-04-16T05:34:25.051 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: using public_addr v2:172.21.15.122:0/0 -> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-04-16T05:34:25.051 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: starting mon.smithi122 rank -1 at public addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] at bind addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi122 fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:34:25.051 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: mon.smithi122@-1(???) e0 preinit fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:34:25.051 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 bash[35976]: 5df19eab6d214c734d17d4f0a3c4f2c1cdf7e7b05541c0171eb1bdf91fd855bd 2024-04-16T05:34:25.052 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 systemd[1]: Started Ceph mon.smithi122 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968. 2024-04-16T05:34:25.052 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: mon.smithi122@-1(synchronizing).mds e1 new map 2024-04-16T05:34:25.052 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: mon.smithi122@-1(synchronizing).mds e1 print_map 2024-04-16T05:34:25.052 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: e1 2024-04-16T05:34:25.052 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:34:25.052 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:34:25.052 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: legacy client fscid: -1 2024-04-16T05:34:25.052 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-16T05:34:25.052 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: No filesystems configured 2024-04-16T05:34:25.052 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: mon.smithi122@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-16T05:34:25.052 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: mon.smithi122@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-16T05:34:25.052 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: mon.smithi122@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-16T05:34:25.052 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: mon.smithi122@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-16T05:34:25.052 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: mon.smithi122@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-16T05:34:25.052 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: mon.smithi122@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-16T05:34:25.052 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: mon.smithi122@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-04-16T05:34:25.053 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: mon.smithi122@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-16T05:34:25.053 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: mon.smithi122@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-16T05:34:25.053 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: mon.smithi122@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-16T05:34:25.053 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: mon.smithi122@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-16T05:34:25.053 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:34:25.053 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:34:25.053 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.053 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:34:25.053 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.qgzypi/mirror_snapshot_schedule"}]: dispatch 2024-04-16T05:34:25.053 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.qgzypi/trash_purge_schedule"}]: dispatch 2024-04-16T05:34:25.053 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.053 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.053 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: mgrmap e16: smithi027.qgzypi(active, since 1.01048s) 2024-04-16T05:34:25.053 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: [16/Apr/2024:05:33:49] ENGINE Bus STARTING 2024-04-16T05:34:25.053 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: [16/Apr/2024:05:33:50] ENGINE Serving on https://172.21.15.27:7150 2024-04-16T05:34:25.053 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: [16/Apr/2024:05:33:50] ENGINE Bus STARTED 2024-04-16T05:34:25.054 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: mgrmap e17: smithi027.qgzypi(active, since 2s) 2024-04-16T05:34:25.054 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.054 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='client.? 172.21.15.122:0/1859621384' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:34:25.054 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.054 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.054 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.054 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.054 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.054 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd/host:smithi027", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:34:25.054 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.054 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.054 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.055 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.055 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:34:25.055 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:34:25.055 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:34:25.055 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='client.? 172.21.15.122:0/1216546504' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:34:25.055 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-16T05:34:25.055 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-16T05:34:25.055 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: Updating smithi122:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:34:25.055 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: Updating smithi027:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:34:25.055 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-16T05:34:25.055 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-16T05:34:25.055 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: Updating smithi122:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T05:34:25.055 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: Updating smithi027:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T05:34:25.055 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.056 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.056 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.056 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.056 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.056 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T05:34:25.056 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-16T05:34:25.056 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:34:25.056 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: Deploying daemon crash.smithi122 on smithi122 2024-04-16T05:34:25.056 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='client.? 172.21.15.122:0/3864378153' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:34:25.056 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:34:25.056 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.056 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.056 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.056 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.056 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: Deploying daemon node-exporter.smithi122 on smithi122 2024-04-16T05:34:25.057 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='client.? 172.21.15.122:0/3015007146' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:34:25.057 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:25.057 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.057 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:25.057 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.057 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.057 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.057 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.057 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.sgokkq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T05:34:25.057 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.sgokkq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-16T05:34:25.057 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T05:34:25.057 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:34:25.057 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: Deploying daemon mgr.smithi122.sgokkq on smithi122 2024-04-16T05:34:25.057 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='client.? 172.21.15.122:0/3834100744' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:34:25.057 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:25.057 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.058 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:25.058 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:25.058 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.058 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.058 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.058 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.058 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T05:34:25.058 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:34:25.059 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='client.? 172.21.15.122:0/3605635787' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:34:25.059 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: Deploying daemon mon.smithi122 on smithi122 2024-04-16T05:34:25.059 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:34:25.059 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:25.059 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:25.059 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:25.059 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: from='client.? 172.21.15.122:0/2993808960' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:34:25.060 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:25.060 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:24 smithi122 ceph-mon[35998]: mon.smithi122@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-04-16T05:34:25.060 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-16T05:34:25.060 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-16T05:34:25.061 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:24 smithi027 ceph-mon[29206]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:25.509 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi122/config 2024-04-16T05:34:30.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-16T05:34:30.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: mon.smithi027 calling monitor election 2024-04-16T05:34:30.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:34:30.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:30.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:34:30.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.122:0/2400073909' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.sgokkq/crt"}]: dispatch 2024-04-16T05:34:30.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:34:30.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: mon.smithi122 calling monitor election 2024-04-16T05:34:30.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:30.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:34:30.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:34:30.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:30.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:34:30.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: mon.smithi027 is new leader, mons smithi027,smithi122 in quorum (ranks 0,1) 2024-04-16T05:34:30.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: monmap e2: 2 mons at {smithi027=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],smithi122=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-16T05:34:30.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: fsmap 2024-04-16T05:34:30.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: osdmap e5: 0 total, 0 up, 0 in 2024-04-16T05:34:30.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: mgrmap e17: smithi027.qgzypi(active, since 40s) 2024-04-16T05:34:30.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: overall HEALTH_OK 2024-04-16T05:34:30.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: Standby manager daemon smithi122.sgokkq started 2024-04-16T05:34:30.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.122:0/2400073909' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T05:34:30.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:30.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.122:0/2400073909' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.sgokkq/key"}]: dispatch 2024-04-16T05:34:30.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.122:0/2400073909' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T05:34:30.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:30.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:30.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-16T05:34:30.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: mon.smithi027 calling monitor election 2024-04-16T05:34:30.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:34:30.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:30.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:34:30.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.122:0/2400073909' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.sgokkq/crt"}]: dispatch 2024-04-16T05:34:30.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:34:30.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: mon.smithi122 calling monitor election 2024-04-16T05:34:30.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:30.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:34:30.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:34:30.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:30.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:34:30.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: mon.smithi027 is new leader, mons smithi027,smithi122 in quorum (ranks 0,1) 2024-04-16T05:34:30.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: monmap e2: 2 mons at {smithi027=[v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0],smithi122=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-04-16T05:34:30.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: fsmap 2024-04-16T05:34:30.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: osdmap e5: 0 total, 0 up, 0 in 2024-04-16T05:34:30.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: mgrmap e17: smithi027.qgzypi(active, since 40s) 2024-04-16T05:34:30.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: overall HEALTH_OK 2024-04-16T05:34:30.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: Standby manager daemon smithi122.sgokkq started 2024-04-16T05:34:30.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.122:0/2400073909' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T05:34:30.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:30.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.122:0/2400073909' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.sgokkq/key"}]: dispatch 2024-04-16T05:34:30.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.122:0/2400073909' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T05:34:30.310 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:30.310 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:30.498 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.499+0000 7fa633200700 1 -- 172.21.15.122:0/2873163182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa610003580 msgr2=0x7fa610005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:30.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.499+0000 7fa633200700 1 --2- 172.21.15.122:0/2873163182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa610003580 0x7fa610005a00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa62c0767e0 tx=0x7fa61c009040 comp rx=0 tx=0).stop 2024-04-16T05:34:30.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.499+0000 7fa633200700 1 -- 172.21.15.122:0/2873163182 shutdown_connections 2024-04-16T05:34:30.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.499+0000 7fa633200700 1 --2- 172.21.15.122:0/2873163182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa610003580 0x7fa610005a00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:30.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.499+0000 7fa633200700 1 -- 172.21.15.122:0/2873163182 >> 172.21.15.122:0/2873163182 conn(0x7fa62c0729d0 msgr2=0x7fa62c074de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:34:30.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.499+0000 7fa633200700 1 -- 172.21.15.122:0/2873163182 shutdown_connections 2024-04-16T05:34:30.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.500+0000 7fa633200700 1 -- 172.21.15.122:0/2873163182 wait complete. 2024-04-16T05:34:30.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.501+0000 7fa633200700 1 Processor -- start 2024-04-16T05:34:30.500 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.502+0000 7fa633200700 1 -- start start 2024-04-16T05:34:30.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.502+0000 7fa633200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa610003580 0x7fa62c119920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:34:30.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.502+0000 7fa633200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62c119e60 0x7fa62c1138a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:34:30.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.502+0000 7fa633200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c11a420 con 0x7fa62c119e60 2024-04-16T05:34:30.501 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.502+0000 7fa633200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c113de0 con 0x7fa610003580 2024-04-16T05:34:30.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.503+0000 7fa623400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62c119e60 0x7fa62c1138a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:34:30.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.503+0000 7fa623400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62c119e60 0x7fa62c1138a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.122:52552/0 (socket says 172.21.15.122:52552) 2024-04-16T05:34:30.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.503+0000 7fa623400700 1 -- 172.21.15.122:0/1116556345 learned_addr learned my addr 172.21.15.122:0/1116556345 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:34:30.502 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.503+0000 7fa62be00700 1 --2- 172.21.15.122:0/1116556345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa610003580 0x7fa62c119920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:34:30.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.504+0000 7fa62be00700 1 -- 172.21.15.122:0/1116556345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa610003580 msgr2=0x7fa62c119920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-16T05:34:30.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.504+0000 7fa62be00700 1 -- 172.21.15.122:0/1116556345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa610003580 msgr2=0x7fa62c119920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-16T05:34:30.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.504+0000 7fa62be00700 1 --2- 172.21.15.122:0/1116556345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa610003580 0x7fa62c119920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-16T05:34:30.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.504+0000 7fa62be00700 1 --2- 172.21.15.122:0/1116556345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa610003580 0x7fa62c119920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T05:34:30.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.504+0000 7fa623400700 1 -- 172.21.15.122:0/1116556345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa610003580 msgr2=0x7fa62c119920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T05:34:30.503 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.504+0000 7fa623400700 1 --2- 172.21.15.122:0/1116556345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa610003580 0x7fa62c119920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:30.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.504+0000 7fa623400700 1 -- 172.21.15.122:0/1116556345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa61c0079e0 con 0x7fa62c119e60 2024-04-16T05:34:30.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.504+0000 7fa623400700 1 --2- 172.21.15.122:0/1116556345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62c119e60 0x7fa62c1138a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa61400b790 tx=0x7fa614009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:34:30.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.505+0000 7fa628c00700 1 -- 172.21.15.122:0/1116556345 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa614010870 con 0x7fa62c119e60 2024-04-16T05:34:30.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.505+0000 7fa633200700 1 -- 172.21.15.122:0/1116556345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa62c1140c0 con 0x7fa62c119e60 2024-04-16T05:34:30.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.505+0000 7fa628c00700 1 -- 172.21.15.122:0/1116556345 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa61400f3e0 con 0x7fa62c119e60 2024-04-16T05:34:30.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.505+0000 7fa628c00700 1 -- 172.21.15.122:0/1116556345 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa614010ea0 con 0x7fa62c119e60 2024-04-16T05:34:30.504 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.505+0000 7fa633200700 1 -- 172.21.15.122:0/1116556345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa62c114680 con 0x7fa62c119e60 2024-04-16T05:34:30.505 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.507+0000 7fa633200700 1 -- 172.21.15.122:0/1116556345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa62c10d730 con 0x7fa62c119e60 2024-04-16T05:34:30.506 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.508+0000 7fa628c00700 1 -- 172.21.15.122:0/1116556345 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86573+0+0 (secure 0 0 0) 0x7fa61400fa30 con 0x7fa62c119e60 2024-04-16T05:34:30.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.509+0000 7fa628c00700 1 --2- 172.21.15.122:0/1116556345 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa608069360 0x7fa60806b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:34:30.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.509+0000 7fa628c00700 1 -- 172.21.15.122:0/1116556345 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fa614085fa0 con 0x7fa62c119e60 2024-04-16T05:34:30.508 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.509+0000 7fa62be00700 1 --2- 172.21.15.122:0/1116556345 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa608069360 0x7fa60806b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:34:30.509 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.511+0000 7fa62be00700 1 --2- 172.21.15.122:0/1116556345 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa608069360 0x7fa60806b810 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa61c004bc0 tx=0x7fa61c009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:34:30.521 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.522+0000 7fa628c00700 1 -- 172.21.15.122:0/1116556345 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa614056e20 con 0x7fa62c119e60 2024-04-16T05:34:30.847 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.848+0000 7fa633200700 1 -- 172.21.15.122:0/1116556345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa62c1048f0 con 0x7fa62c119e60 2024-04-16T05:34:30.848 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.849+0000 7fa628c00700 1 -- 172.21.15.122:0/1116556345 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1017 (secure 0 0 0) 0x7fa6140569b0 con 0x7fa62c119e60 2024-04-16T05:34:30.848 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-04-16T05:34:30.848 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":2,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","modified":"2024-04-16T05:34:24.745534Z","created":"2024-04-16T05:30:56.860193Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi027","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:3300","nonce":0},{"type":"v1","addr":"172.21.15.27:6789","nonce":0}]},"addr":"172.21.15.27:6789/0","public_addr":"172.21.15.27:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi122","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-16T05:34:30.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.853+0000 7fa633200700 1 -- 172.21.15.122:0/1116556345 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa608069360 msgr2=0x7fa60806b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:30.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.853+0000 7fa633200700 1 --2- 172.21.15.122:0/1116556345 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa608069360 0x7fa60806b810 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa61c004bc0 tx=0x7fa61c009040 comp rx=0 tx=0).stop 2024-04-16T05:34:30.852 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.853+0000 7fa633200700 1 -- 172.21.15.122:0/1116556345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62c119e60 msgr2=0x7fa62c1138a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:30.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.853+0000 7fa633200700 1 --2- 172.21.15.122:0/1116556345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62c119e60 0x7fa62c1138a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa61400b790 tx=0x7fa614009250 comp rx=0 tx=0).stop 2024-04-16T05:34:30.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.853+0000 7fa633200700 1 -- 172.21.15.122:0/1116556345 shutdown_connections 2024-04-16T05:34:30.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.853+0000 7fa633200700 1 --2- 172.21.15.122:0/1116556345 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa608069360 0x7fa60806b810 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:30.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.853+0000 7fa633200700 1 --2- 172.21.15.122:0/1116556345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa610003580 0x7fa62c119920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:30.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.853+0000 7fa633200700 1 --2- 172.21.15.122:0/1116556345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa62c119e60 0x7fa62c1138a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:30.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.853+0000 7fa633200700 1 -- 172.21.15.122:0/1116556345 >> 172.21.15.122:0/1116556345 conn(0x7fa62c0729d0 msgr2=0x7fa62c108580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:34:30.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.854+0000 7fa633200700 1 -- 172.21.15.122:0/1116556345 shutdown_connections 2024-04-16T05:34:30.853 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:34:30.854+0000 7fa633200700 1 -- 172.21.15.122:0/1116556345 wait complete. 2024-04-16T05:34:30.853 INFO:teuthology.orchestra.run.smithi122.stderr:dumped monmap epoch 2 2024-04-16T05:34:31.222 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:30 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:31.222 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:30 smithi122 ceph-mon[35998]: mgrmap e18: smithi027.qgzypi(active, since 40s), standbys: smithi122.sgokkq 2024-04-16T05:34:31.222 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:30 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr metadata", "who": "smithi122.sgokkq", "id": "smithi122.sgokkq"}]: dispatch 2024-04-16T05:34:31.222 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:30 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:31.222 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:30 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:34:31.222 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:30 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:34:31.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:30 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:31.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:30 smithi027 ceph-mon[29206]: mgrmap e18: smithi027.qgzypi(active, since 40s), standbys: smithi122.sgokkq 2024-04-16T05:34:31.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:30 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr metadata", "who": "smithi122.sgokkq", "id": "smithi122.sgokkq"}]: dispatch 2024-04-16T05:34:31.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:30 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:31.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:30 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:34:31.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:30 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:34:31.662 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-16T05:34:31.663 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph config generate-minimal-conf 2024-04-16T05:34:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:31 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/1116556345' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:34:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:31 smithi027 ceph-mon[29206]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:32.191 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:34:32.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:31 smithi122 ceph-mon[35998]: from='client.? 172.21.15.122:0/1116556345' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-16T05:34:32.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:31 smithi122 ceph-mon[35998]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.313+0000 7f4e07e00700 1 -- 172.21.15.27:0/2393867767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e00071f90 msgr2=0x7f4e00070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.313+0000 7f4e07e00700 1 --2- 172.21.15.27:0/2393867767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e00071f90 0x7f4e00070360 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4df40048c0 tx=0x7f4df4009e30 comp rx=0 tx=0).stop 2024-04-16T05:34:34.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.313+0000 7f4e07e00700 1 -- 172.21.15.27:0/2393867767 shutdown_connections 2024-04-16T05:34:34.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.313+0000 7f4e07e00700 1 --2- 172.21.15.27:0/2393867767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e00071f90 0x7f4e00070360 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:34.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.313+0000 7f4e07e00700 1 -- 172.21.15.27:0/2393867767 >> 172.21.15.27:0/2393867767 conn(0x7f4e000f85f0 msgr2=0x7f4e000faa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:34:34.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.314+0000 7f4e07e00700 1 -- 172.21.15.27:0/2393867767 shutdown_connections 2024-04-16T05:34:34.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.314+0000 7f4e07e00700 1 -- 172.21.15.27:0/2393867767 wait complete. 2024-04-16T05:34:34.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.314+0000 7f4e07e00700 1 Processor -- start 2024-04-16T05:34:34.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.315+0000 7f4e07e00700 1 -- start start 2024-04-16T05:34:34.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.315+0000 7f4e07e00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e00071f90 0x7f4e00119940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:34:34.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.315+0000 7f4e07e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e00119e80 0x7f4e001138c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:34:34.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.315+0000 7f4e07e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0011a440 con 0x7f4e00119e80 2024-04-16T05:34:34.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.315+0000 7f4e07e00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e00113e00 con 0x7f4e00071f90 2024-04-16T05:34:34.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.315+0000 7f4e04a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e00119e80 0x7f4e001138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:34:34.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.315+0000 7f4e04a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e00119e80 0x7f4e001138c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52748/0 (socket says 172.21.15.27:52748) 2024-04-16T05:34:34.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.316+0000 7f4e04a00700 1 -- 172.21.15.27:0/1909314647 learned_addr learned my addr 172.21.15.27:0/1909314647 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:34:34.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.316+0000 7f4e05400700 1 --2- 172.21.15.27:0/1909314647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e00071f90 0x7f4e00119940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:34:34.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.316+0000 7f4e05400700 1 -- 172.21.15.27:0/1909314647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e00071f90 msgr2=0x7f4e00119940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-16T05:34:34.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.316+0000 7f4e05400700 1 -- 172.21.15.27:0/1909314647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e00071f90 msgr2=0x7f4e00119940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-16T05:34:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.316+0000 7f4e05400700 1 --2- 172.21.15.27:0/1909314647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e00071f90 0x7f4e00119940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-16T05:34:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.316+0000 7f4e04a00700 1 -- 172.21.15.27:0/1909314647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e00071f90 msgr2=0x7f4e00119940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.316+0000 7f4e05400700 1 --2- 172.21.15.27:0/1909314647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e00071f90 0x7f4e00119940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T05:34:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.316+0000 7f4e04a00700 1 --2- 172.21.15.27:0/1909314647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e00071f90 0x7f4e00119940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.316+0000 7f4e04a00700 1 -- 172.21.15.27:0/1909314647 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df4005040 con 0x7f4e00119e80 2024-04-16T05:34:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.317+0000 7f4e04a00700 1 --2- 172.21.15.27:0/1909314647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e00119e80 0x7f4e001138c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00b790 tx=0x7f4dec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:34:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.317+0000 7f4dfd600700 1 -- 172.21.15.27:0/1909314647 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4dec010870 con 0x7f4e00119e80 2024-04-16T05:34:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.317+0000 7f4dfd600700 1 -- 172.21.15.27:0/1909314647 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4dec00f3e0 con 0x7f4e00119e80 2024-04-16T05:34:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.317+0000 7f4e07e00700 1 -- 172.21.15.27:0/1909314647 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e001140e0 con 0x7f4e00119e80 2024-04-16T05:34:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.317+0000 7f4dfd600700 1 -- 172.21.15.27:0/1909314647 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4dec010ea0 con 0x7f4e00119e80 2024-04-16T05:34:34.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.317+0000 7f4e07e00700 1 -- 172.21.15.27:0/1909314647 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e00114620 con 0x7f4e00119e80 2024-04-16T05:34:34.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.319+0000 7f4e07e00700 1 -- 172.21.15.27:0/1909314647 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e000fff40 con 0x7f4e00119e80 2024-04-16T05:34:34.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.320+0000 7f4dfd600700 1 -- 172.21.15.27:0/1909314647 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86573+0+0 (secure 0 0 0) 0x7f4dec00f550 con 0x7f4e00119e80 2024-04-16T05:34:34.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.322+0000 7f4dfd600700 1 --2- 172.21.15.27:0/1909314647 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f4de8069360 0x7f4de806b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:34:34.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.322+0000 7f4dfd600700 1 -- 172.21.15.27:0/1909314647 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f4dec0865f0 con 0x7f4e00119e80 2024-04-16T05:34:34.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.323+0000 7f4e05400700 1 --2- 172.21.15.27:0/1909314647 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f4de8069360 0x7f4de806b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:34:34.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.324+0000 7f4e05400700 1 --2- 172.21.15.27:0/1909314647 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f4de8069360 0x7f4de806b810 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4df400b3b0 tx=0x7f4df4009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:34:34.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.326+0000 7f4dfd600700 1 -- 172.21.15.27:0/1909314647 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f4dec057490 con 0x7f4e00119e80 2024-04-16T05:34:34.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.574+0000 7f4e07e00700 1 -- 172.21.15.27:0/1909314647 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f4e0010d3d0 con 0x7f4e00119e80 2024-04-16T05:34:34.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.576+0000 7f4dfd600700 1 -- 172.21.15.27:0/1909314647 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7f4dec014100 con 0x7f4e00119e80 2024-04-16T05:34:34.575 INFO:teuthology.orchestra.run.smithi027.stdout:# minimal ceph.conf for 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:34:34.576 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-04-16T05:34:34.576 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:34:34.576 INFO:teuthology.orchestra.run.smithi027.stdout: mon_host = [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2024-04-16T05:34:34.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.579+0000 7f4e07e00700 1 -- 172.21.15.27:0/1909314647 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f4de8069360 msgr2=0x7f4de806b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:34.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.579+0000 7f4e07e00700 1 --2- 172.21.15.27:0/1909314647 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f4de8069360 0x7f4de806b810 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4df400b3b0 tx=0x7f4df4009e30 comp rx=0 tx=0).stop 2024-04-16T05:34:34.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.579+0000 7f4e07e00700 1 -- 172.21.15.27:0/1909314647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e00119e80 msgr2=0x7f4e001138c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:34.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.579+0000 7f4e07e00700 1 --2- 172.21.15.27:0/1909314647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e00119e80 0x7f4e001138c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00b790 tx=0x7f4dec009250 comp rx=0 tx=0).stop 2024-04-16T05:34:34.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.580+0000 7f4e07e00700 1 -- 172.21.15.27:0/1909314647 shutdown_connections 2024-04-16T05:34:34.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.580+0000 7f4e07e00700 1 --2- 172.21.15.27:0/1909314647 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f4de8069360 0x7f4de806b810 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:34.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.580+0000 7f4e07e00700 1 --2- 172.21.15.27:0/1909314647 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e00071f90 0x7f4e00119940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:34.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.580+0000 7f4e07e00700 1 --2- 172.21.15.27:0/1909314647 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e00119e80 0x7f4e001138c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:34.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.580+0000 7f4e07e00700 1 -- 172.21.15.27:0/1909314647 >> 172.21.15.27:0/1909314647 conn(0x7f4e000f85f0 msgr2=0x7f4e000fa4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:34:34.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.580+0000 7f4e07e00700 1 -- 172.21.15.27:0/1909314647 shutdown_connections 2024-04-16T05:34:34.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:34.580+0000 7f4e07e00700 1 -- 172.21.15.27:0/1909314647 wait complete. 2024-04-16T05:34:34.737 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:34 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:34.738 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:34 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:34.738 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:34 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:34:34.738 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:34 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:34:34.738 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:34 smithi027 ceph-mon[29206]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-16T05:34:34.738 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:34 smithi027 ceph-mon[29206]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-16T05:34:34.738 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:34 smithi027 ceph-mon[29206]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:34.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:34 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:34.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:34 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:34.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:34 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:34:34.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:34 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:34:34.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:34 smithi122 ceph-mon[35998]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-16T05:34:34.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:34 smithi122 ceph-mon[35998]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-16T05:34:34.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:34 smithi122 ceph-mon[35998]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:35.383 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-16T05:34:35.383 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-16T05:34:35.383 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.conf 2024-04-16T05:34:35.464 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-16T05:34:35.464 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-16T05:34:35.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:35 smithi027 ceph-mon[29206]: Updating smithi027:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:34:35.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:35 smithi027 ceph-mon[29206]: Updating smithi122:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:34:35.587 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:34:35.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:35.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:35 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1909314647' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:34:35.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:35.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:35.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:35.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:35.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:35.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T05:34:35.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T05:34:35.588 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:34:35.601 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-16T05:34:35.601 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.conf 2024-04-16T05:34:35.702 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-16T05:34:35.702 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-16T05:34:35.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:35 smithi122 ceph-mon[35998]: Updating smithi027:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:34:35.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:35 smithi122 ceph-mon[35998]: Updating smithi122:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:34:35.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:34:35.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:35.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:35 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/1909314647' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:34:35.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:35.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:35.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:35.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:35.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:35.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T05:34:35.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T05:34:35.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:34:35.840 INFO:tasks.cephadm:Deploying OSDs... 2024-04-16T05:34:35.840 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-16T05:34:35.840 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T05:34:35.891 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T05:34:35.891 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2024-04-16T05:34:35.983 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T05:34:35.983 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T05:34:35.983 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 633 Links: 1 2024-04-16T05:34:35.983 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T05:34:35.983 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T05:34:35.983 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-16 05:33:38.333383171 +0000 2024-04-16T05:34:35.983 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-16 05:26:13.661304834 +0000 2024-04-16T05:34:35.983 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-16 05:26:13.661304834 +0000 2024-04-16T05:34:35.983 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-16 05:26:13.661304834 +0000 2024-04-16T05:34:35.983 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T05:34:36.076 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-16T05:34:36.076 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-16T05:34:36.077 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000376985 s, 1.4 MB/s 2024-04-16T05:34:36.080 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T05:34:36.136 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2024-04-16T05:34:36.179 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T05:34:36.179 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T05:34:36.179 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 651 Links: 1 2024-04-16T05:34:36.179 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T05:34:36.179 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T05:34:36.179 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-16 05:33:38.333383171 +0000 2024-04-16T05:34:36.179 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-16 05:26:14.324287935 +0000 2024-04-16T05:34:36.179 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-16 05:26:14.324287935 +0000 2024-04-16T05:34:36.180 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-16 05:26:14.324287935 +0000 2024-04-16T05:34:36.180 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T05:34:36.320 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-16T05:34:36.320 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-16T05:34:36.320 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.00035279 s, 1.5 MB/s 2024-04-16T05:34:36.325 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T05:34:36.378 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2024-04-16T05:34:36.427 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T05:34:36.427 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T05:34:36.427 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 671 Links: 1 2024-04-16T05:34:36.427 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T05:34:36.428 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T05:34:36.428 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-16 05:33:38.334383146 +0000 2024-04-16T05:34:36.428 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-16 05:26:14.937272311 +0000 2024-04-16T05:34:36.428 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-16 05:26:14.937272311 +0000 2024-04-16T05:34:36.428 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-16 05:26:14.937272311 +0000 2024-04-16T05:34:36.428 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T05:34:36.516 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-16T05:34:36.516 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-16T05:34:36.516 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000273537 s, 1.9 MB/s 2024-04-16T05:34:36.520 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T05:34:36.565 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2024-04-16T05:34:36.661 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T05:34:36.661 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T05:34:36.661 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 689 Links: 1 2024-04-16T05:34:36.661 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T05:34:36.661 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T05:34:36.661 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-04-16 05:33:38.335383120 +0000 2024-04-16T05:34:36.661 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-04-16 05:26:15.589255692 +0000 2024-04-16T05:34:36.661 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-04-16 05:26:15.589255692 +0000 2024-04-16T05:34:36.661 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-04-16 05:26:15.589255692 +0000 2024-04-16T05:34:36.661 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T05:34:36.750 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:36 smithi027 ceph-mon[29206]: Reconfiguring mon.smithi027 (unknown last config time)... 2024-04-16T05:34:36.751 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:36 smithi027 ceph-mon[29206]: Reconfiguring daemon mon.smithi027 on smithi027 2024-04-16T05:34:36.751 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:36 smithi027 ceph-mon[29206]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:36.759 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-04-16T05:34:36.759 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-04-16T05:34:36.759 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000324821 s, 1.6 MB/s 2024-04-16T05:34:36.763 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T05:34:36.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:36 smithi122 ceph-mon[35998]: Reconfiguring mon.smithi027 (unknown last config time)... 2024-04-16T05:34:36.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:36 smithi122 ceph-mon[35998]: Reconfiguring daemon mon.smithi027 on smithi027 2024-04-16T05:34:36.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:36 smithi122 ceph-mon[35998]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:36.823 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-16T05:34:36.823 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T05:34:36.880 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T05:34:36.880 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_1 2024-04-16T05:34:36.932 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T05:34:36.932 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T05:34:36.932 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 631 Links: 1 2024-04-16T05:34:36.932 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T05:34:36.932 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T05:34:36.932 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-16 05:34:23.456847080 +0000 2024-04-16T05:34:36.932 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-16 05:25:51.427973175 +0000 2024-04-16T05:34:36.932 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-16 05:25:51.427973175 +0000 2024-04-16T05:34:36.932 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-04-16 05:25:51.427973175 +0000 2024-04-16T05:34:36.932 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T05:34:37.018 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-16T05:34:37.018 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-16T05:34:37.018 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000608956 s, 841 kB/s 2024-04-16T05:34:37.021 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T05:34:37.076 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_2 2024-04-16T05:34:37.125 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T05:34:37.125 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T05:34:37.125 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 653 Links: 1 2024-04-16T05:34:37.125 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T05:34:37.125 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T05:34:37.125 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-16 05:34:23.457847056 +0000 2024-04-16T05:34:37.125 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-16 05:25:51.994960127 +0000 2024-04-16T05:34:37.125 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-16 05:25:51.994960127 +0000 2024-04-16T05:34:37.125 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-04-16 05:25:51.994960127 +0000 2024-04-16T05:34:37.125 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T05:34:37.209 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-16T05:34:37.209 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-16T05:34:37.209 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000332553 s, 1.5 MB/s 2024-04-16T05:34:37.213 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T05:34:37.269 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_3 2024-04-16T05:34:37.317 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T05:34:37.318 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T05:34:37.318 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 670 Links: 1 2024-04-16T05:34:37.318 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T05:34:37.318 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T05:34:37.318 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-16 05:34:23.458847033 +0000 2024-04-16T05:34:37.318 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-16 05:25:52.584946549 +0000 2024-04-16T05:34:37.318 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-16 05:25:52.585946526 +0000 2024-04-16T05:34:37.318 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-04-16 05:25:52.584946549 +0000 2024-04-16T05:34:37.318 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T05:34:37.398 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-16T05:34:37.398 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-16T05:34:37.398 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000321319 s, 1.6 MB/s 2024-04-16T05:34:37.401 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T05:34:37.455 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_4 2024-04-16T05:34:37.503 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T05:34:37.503 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T05:34:37.503 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 688 Links: 1 2024-04-16T05:34:37.503 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T05:34:37.503 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T05:34:37.503 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-04-16 05:34:23.458847033 +0000 2024-04-16T05:34:37.504 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-04-16 05:25:53.218931959 +0000 2024-04-16T05:34:37.504 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-04-16 05:25:53.218931959 +0000 2024-04-16T05:34:37.504 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-04-16 05:25:53.218931959 +0000 2024-04-16T05:34:37.504 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T05:34:37.586 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-04-16T05:34:37.586 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-04-16T05:34:37.586 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000326328 s, 1.6 MB/s 2024-04-16T05:34:37.589 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T05:34:37.644 INFO:tasks.cephadm:Deploying osd.0 on smithi027 with /dev/vg_nvme/lv_4... 2024-04-16T05:34:37.645 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-16T05:34:38.128 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:34:38.676 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:38 smithi027 ceph-mon[29206]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:38.676 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:38 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:38.676 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:38 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:38.676 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:38 smithi027 ceph-mon[29206]: Reconfiguring mgr.smithi027.qgzypi (unknown last config time)... 2024-04-16T05:34:38.676 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:38 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi027.qgzypi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T05:34:38.676 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:38 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T05:34:38.676 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:38 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:34:38.676 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:38 smithi027 ceph-mon[29206]: Reconfiguring daemon mgr.smithi027.qgzypi on smithi027 2024-04-16T05:34:38.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:38 smithi122 ceph-mon[35998]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:38.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:38 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:38.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:38 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:38.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:38 smithi122 ceph-mon[35998]: Reconfiguring mgr.smithi027.qgzypi (unknown last config time)... 2024-04-16T05:34:38.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:38 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi027.qgzypi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T05:34:38.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:38 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T05:34:38.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:38 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:34:38.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:38 smithi122 ceph-mon[35998]: Reconfiguring daemon mgr.smithi027.qgzypi on smithi027 2024-04-16T05:34:40.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:40 smithi122 ceph-mon[35998]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:40.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:40 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:40.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:40 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:40.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:40 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi027", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T05:34:40.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:40 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:34:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:40 smithi027 ceph-mon[29206]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:40 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:40 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:40 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi027", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T05:34:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:40 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:34:41.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:41 smithi122 ceph-mon[35998]: Reconfiguring crash.smithi027 (monmap changed)... 2024-04-16T05:34:41.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:41 smithi122 ceph-mon[35998]: Reconfiguring daemon crash.smithi027 on smithi027 2024-04-16T05:34:41.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:41 smithi027 ceph-mon[29206]: Reconfiguring crash.smithi027 (monmap changed)... 2024-04-16T05:34:41.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:41 smithi027 ceph-mon[29206]: Reconfiguring daemon crash.smithi027 on smithi027 2024-04-16T05:34:42.069 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-16T05:34:42.069 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-16T05:34:42.069 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-16T05:34:42.070 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-16T05:34:42.070 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0516277 s, 203 MB/s 2024-04-16T05:34:42.070 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-16T05:34:42.635 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:42 smithi027 ceph-mon[29206]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:43.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:42 smithi122 ceph-mon[35998]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:43.048 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_4 2024-04-16T05:34:43.618 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:34:44.966 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:44 smithi027 ceph-mon[29206]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:44.966 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:44 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:44.967 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:44 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:44.967 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:44 smithi027 ceph-mon[29206]: Reconfiguring alertmanager.smithi027 (dependencies changed)... 2024-04-16T05:34:44.967 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:44 smithi027 ceph-mon[29206]: Reconfiguring daemon alertmanager.smithi027 on smithi027 2024-04-16T05:34:45.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:44 smithi122 ceph-mon[35998]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:45.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:44 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:45.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:44 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:45.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:44 smithi122 ceph-mon[35998]: Reconfiguring alertmanager.smithi027 (dependencies changed)... 2024-04-16T05:34:45.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:44 smithi122 ceph-mon[35998]: Reconfiguring daemon alertmanager.smithi027 on smithi027 2024-04-16T05:34:46.854 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:46 smithi027 ceph-mon[29206]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:47.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:46 smithi122 ceph-mon[35998]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.475+0000 7f82fe200700 1 -- 172.21.15.27:0/3325520931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f81002d0 msgr2=0x7f82f8100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.475+0000 7f82fe200700 1 --2- 172.21.15.27:0/3325520931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f81002d0 0x7f82f8100740 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f82e4004940 tx=0x7f82e4009e30 comp rx=0 tx=0).stop 2024-04-16T05:34:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.475+0000 7f82fe200700 1 -- 172.21.15.27:0/3325520931 shutdown_connections 2024-04-16T05:34:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.475+0000 7f82fe200700 1 --2- 172.21.15.27:0/3325520931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f81002d0 0x7f82f8100740 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.475+0000 7f82fe200700 1 --2- 172.21.15.27:0/3325520931 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82f80ff020 0x7f82f80ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.475+0000 7f82fe200700 1 -- 172.21.15.27:0/3325520931 >> 172.21.15.27:0/3325520931 conn(0x7f82f80fa950 msgr2=0x7f82f80fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:34:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.475+0000 7f82fe200700 1 -- 172.21.15.27:0/3325520931 shutdown_connections 2024-04-16T05:34:47.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.475+0000 7f82fe200700 1 -- 172.21.15.27:0/3325520931 wait complete. 2024-04-16T05:34:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.476+0000 7f82fe200700 1 Processor -- start 2024-04-16T05:34:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.476+0000 7f82fe200700 1 -- start start 2024-04-16T05:34:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.477+0000 7f82fe200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82f80ff020 0x7f82f810fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:34:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.477+0000 7f82fe200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f81002d0 0x7f82f8110070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:34:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.477+0000 7f82fe200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f8110670 con 0x7f82f81002d0 2024-04-16T05:34:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.477+0000 7f82fe200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f81107e0 con 0x7f82f80ff020 2024-04-16T05:34:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.477+0000 7f82f6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f81002d0 0x7f82f8110070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:34:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.477+0000 7f82f6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f81002d0 0x7f82f8110070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52104/0 (socket says 172.21.15.27:52104) 2024-04-16T05:34:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.477+0000 7f82f6a00700 1 -- 172.21.15.27:0/3151041406 learned_addr learned my addr 172.21.15.27:0/3151041406 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:34:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.477+0000 7f82f7400700 1 --2- 172.21.15.27:0/3151041406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82f80ff020 0x7f82f810fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:34:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.477+0000 7f82f6a00700 1 -- 172.21.15.27:0/3151041406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82f80ff020 msgr2=0x7f82f810fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:34:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.477+0000 7f82f6a00700 1 --2- 172.21.15.27:0/3151041406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82f80ff020 0x7f82f810fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:34:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.477+0000 7f82f6a00700 1 -- 172.21.15.27:0/3151041406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e4005040 con 0x7f82f81002d0 2024-04-16T05:34:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.478+0000 7f82f6a00700 1 --2- 172.21.15.27:0/3151041406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f81002d0 0x7f82f8110070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f82e400b3b0 tx=0x7f82e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:34:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.478+0000 7f82ef400700 1 -- 172.21.15.27:0/3151041406 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82e400e040 con 0x7f82f81002d0 2024-04-16T05:34:47.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.478+0000 7f82ef400700 1 -- 172.21.15.27:0/3151041406 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82e4004290 con 0x7f82f81002d0 2024-04-16T05:34:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.478+0000 7f82fe200700 1 -- 172.21.15.27:0/3151041406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f81151e0 con 0x7f82f81002d0 2024-04-16T05:34:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.478+0000 7f82ef400700 1 -- 172.21.15.27:0/3151041406 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82e401c630 con 0x7f82f81002d0 2024-04-16T05:34:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.478+0000 7f82fe200700 1 -- 172.21.15.27:0/3151041406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f81157a0 con 0x7f82f81002d0 2024-04-16T05:34:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.479+0000 7f82fe200700 1 -- 172.21.15.27:0/3151041406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f804c710 con 0x7f82f81002d0 2024-04-16T05:34:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.480+0000 7f82ef400700 1 -- 172.21.15.27:0/3151041406 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86573+0+0 (secure 0 0 0) 0x7f82e400bbf0 con 0x7f82f81002d0 2024-04-16T05:34:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.481+0000 7f82ef400700 1 --2- 172.21.15.27:0/3151041406 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f82dc069210 0x7f82dc06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:34:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.481+0000 7f82ef400700 1 -- 172.21.15.27:0/3151041406 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f82e40874f0 con 0x7f82f81002d0 2024-04-16T05:34:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.482+0000 7f82f7400700 1 --2- 172.21.15.27:0/3151041406 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f82dc069210 0x7f82dc06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:34:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.482+0000 7f82f7400700 1 --2- 172.21.15.27:0/3151041406 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f82dc069210 0x7f82dc06b6c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f82e8007fd0 tx=0x7f82e8009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:34:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.486+0000 7f82ef400700 1 -- 172.21.15.27:0/3151041406 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f82e4058370 con 0x7f82f81002d0 2024-04-16T05:34:47.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:34:47.727+0000 7f82fe200700 1 -- 172.21.15.27:0/3151041406 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f82f8104640 con 0x7f82dc069210 2024-04-16T05:34:49.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:48 smithi122 ceph-mon[35998]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:49.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:48 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T05:34:49.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:48 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T05:34:49.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:48 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:34:49.023 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:48 smithi027 ceph-mon[29206]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:49.023 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:48 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T05:34:49.024 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:48 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T05:34:49.024 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:48 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:34:49.918 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:49 smithi122 ceph-mon[35998]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:34:49.918 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:49 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:34:49.943 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:49 smithi027 ceph-mon[29206]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:34:49.944 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:49 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:34:50.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:50 smithi122 ceph-mon[35998]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:50.957 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:50 smithi027 ceph-mon[29206]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:52.931 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:52 smithi027 ceph-mon[29206]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:53.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:52 smithi122 ceph-mon[35998]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:54.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:54 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:54.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:54 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:54.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:54 smithi027 ceph-mon[29206]: Reconfiguring grafana.smithi027 (dependencies changed)... 2024-04-16T05:34:54.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:54 smithi027 ceph-mon[29206]: Reconfiguring daemon grafana.smithi027 on smithi027 2024-04-16T05:34:54.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:54 smithi027 ceph-mon[29206]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:54.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:54 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:54.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:54 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:34:54.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:54 smithi122 ceph-mon[35998]: Reconfiguring grafana.smithi027 (dependencies changed)... 2024-04-16T05:34:54.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:54 smithi122 ceph-mon[35998]: Reconfiguring daemon grafana.smithi027 on smithi027 2024-04-16T05:34:54.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:54 smithi122 ceph-mon[35998]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:55.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:55 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/2313103522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4dd7e294-f625-4492-8bd3-18beab34b6b0"}]: dispatch 2024-04-16T05:34:55.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:55 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/2313103522' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4dd7e294-f625-4492-8bd3-18beab34b6b0"}]': finished 2024-04-16T05:34:55.310 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:55 smithi027 ceph-mon[29206]: osdmap e6: 1 total, 0 up, 1 in 2024-04-16T05:34:55.311 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:55 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T05:34:55.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:55 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/2313103522' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4dd7e294-f625-4492-8bd3-18beab34b6b0"}]: dispatch 2024-04-16T05:34:55.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:55 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/2313103522' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4dd7e294-f625-4492-8bd3-18beab34b6b0"}]': finished 2024-04-16T05:34:55.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:55 smithi122 ceph-mon[35998]: osdmap e6: 1 total, 0 up, 1 in 2024-04-16T05:34:55.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:55 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T05:34:56.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:56 smithi122 ceph-mon[35998]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:56.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:56 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/1723165851' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T05:34:56.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:56 smithi027 ceph-mon[29206]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:56.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:56 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1723165851' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T05:34:58.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:34:58 smithi027 ceph-mon[29206]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:34:59.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:34:58 smithi122 ceph-mon[35998]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:00.685 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:00 smithi027 ceph-mon[29206]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:00.831 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:00 smithi122 ceph-mon[35998]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:02.898 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:02 smithi027 ceph-mon[29206]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:03.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:02 smithi122 ceph-mon[35998]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:04.784 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:04 smithi027 ceph-mon[29206]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:04.784 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:04 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:35:05.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:04 smithi122 ceph-mon[35998]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:05.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:04 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:35:06.788 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:06 smithi027 ceph-mon[29206]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:07.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:06 smithi122 ceph-mon[35998]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:07.998 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:07 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:07.998 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:07 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:07.998 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:07 smithi027 ceph-mon[29206]: Reconfiguring prometheus.smithi027 (dependencies changed)... 2024-04-16T05:35:07.998 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:07 smithi027 ceph-mon[29206]: Reconfiguring daemon prometheus.smithi027 on smithi027 2024-04-16T05:35:07.999 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:07 smithi027 ceph-mon[29206]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:08.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:07 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:08.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:07 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:08.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:07 smithi122 ceph-mon[35998]: Reconfiguring prometheus.smithi027 (dependencies changed)... 2024-04-16T05:35:08.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:07 smithi122 ceph-mon[35998]: Reconfiguring daemon prometheus.smithi027 on smithi027 2024-04-16T05:35:08.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:07 smithi122 ceph-mon[35998]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:09.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:08 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T05:35:09.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:08 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:35:09.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:08 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T05:35:09.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:08 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:35:10.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:09 smithi027 ceph-mon[29206]: Deploying daemon osd.0 on smithi027 2024-04-16T05:35:10.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:09 smithi027 ceph-mon[29206]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:10.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:09 smithi122 ceph-mon[35998]: Deploying daemon osd.0 on smithi027 2024-04-16T05:35:10.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:09 smithi122 ceph-mon[35998]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:12.625 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:12 smithi027 ceph-mon[29206]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:13.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:12 smithi122 ceph-mon[35998]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:14.725 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:14 smithi027 ceph-mon[29206]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:14 smithi122 ceph-mon[35998]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:16.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:16 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:16.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:16 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:16.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:16 smithi122 ceph-mon[35998]: Reconfiguring crash.smithi122 (monmap changed)... 2024-04-16T05:35:16.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:16 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T05:35:16.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:16 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:35:16.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:16 smithi122 ceph-mon[35998]: Reconfiguring daemon crash.smithi122 on smithi122 2024-04-16T05:35:16.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:16 smithi122 ceph-mon[35998]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:16.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:16 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:16.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:16 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:16.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:16 smithi027 ceph-mon[29206]: Reconfiguring crash.smithi122 (monmap changed)... 2024-04-16T05:35:16.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:16 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T05:35:16.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:16 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:35:16.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:16 smithi027 ceph-mon[29206]: Reconfiguring daemon crash.smithi122 on smithi122 2024-04-16T05:35:16.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:16 smithi027 ceph-mon[29206]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:18.425 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:18 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:18.425 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:18 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:18.425 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:18 smithi122 ceph-mon[35998]: Reconfiguring mgr.smithi122.sgokkq (monmap changed)... 2024-04-16T05:35:18.425 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:18 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.sgokkq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T05:35:18.425 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:18 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T05:35:18.426 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:18 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:35:18.426 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:18 smithi122 ceph-mon[35998]: Reconfiguring daemon mgr.smithi122.sgokkq on smithi122 2024-04-16T05:35:18.426 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:18 smithi122 ceph-mon[35998]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:18.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:18 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:18.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:18 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:18.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:18 smithi027 ceph-mon[29206]: Reconfiguring mgr.smithi122.sgokkq (monmap changed)... 2024-04-16T05:35:18.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:18 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.sgokkq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T05:35:18.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:18 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T05:35:18.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:18 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:35:18.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:18 smithi027 ceph-mon[29206]: Reconfiguring daemon mgr.smithi122.sgokkq on smithi122 2024-04-16T05:35:18.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:18 smithi027 ceph-mon[29206]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:20.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:19 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:20.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:19 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:20.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:19 smithi122 ceph-mon[35998]: Reconfiguring mon.smithi122 (monmap changed)... 2024-04-16T05:35:20.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:19 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T05:35:20.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:19 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T05:35:20.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:19 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:35:20.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:19 smithi122 ceph-mon[35998]: Reconfiguring daemon mon.smithi122 on smithi122 2024-04-16T05:35:20.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:19 smithi122 ceph-mon[35998]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:20.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:19 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:35:20.279 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:19 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:20.279 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:19 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:20.279 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:19 smithi027 ceph-mon[29206]: Reconfiguring mon.smithi122 (monmap changed)... 2024-04-16T05:35:20.280 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:19 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T05:35:20.280 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:19 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T05:35:20.280 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:19 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:35:20.280 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:19 smithi027 ceph-mon[29206]: Reconfiguring daemon mon.smithi122 on smithi122 2024-04-16T05:35:20.280 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:19 smithi027 ceph-mon[29206]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:20.280 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:19 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:35:22.168 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:22.168 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:22.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T05:35:22.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:21 smithi122 ceph-mon[35998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T05:35:22.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi027.front.sepia.ceph.com:9093"}]: dispatch 2024-04-16T05:35:22.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:21 smithi122 ceph-mon[35998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi027.front.sepia.ceph.com:9093"}]: dispatch 2024-04-16T05:35:22.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:22.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T05:35:22.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:21 smithi122 ceph-mon[35998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T05:35:22.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi027.front.sepia.ceph.com:3000"}]: dispatch 2024-04-16T05:35:22.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:21 smithi122 ceph-mon[35998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi027.front.sepia.ceph.com:3000"}]: dispatch 2024-04-16T05:35:22.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:22.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T05:35:22.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:21 smithi122 ceph-mon[35998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T05:35:22.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi027.front.sepia.ceph.com:9095"}]: dispatch 2024-04-16T05:35:22.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:21 smithi122 ceph-mon[35998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi027.front.sepia.ceph.com:9095"}]: dispatch 2024-04-16T05:35:22.169 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:22.170 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:35:22.170 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:21 smithi122 ceph-mon[35998]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:22.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:22.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:22.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T05:35:22.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:21 smithi027 ceph-mon[29206]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T05:35:22.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi027.front.sepia.ceph.com:9093"}]: dispatch 2024-04-16T05:35:22.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:21 smithi027 ceph-mon[29206]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi027.front.sepia.ceph.com:9093"}]: dispatch 2024-04-16T05:35:22.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:22.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T05:35:22.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:21 smithi027 ceph-mon[29206]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T05:35:22.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi027.front.sepia.ceph.com:3000"}]: dispatch 2024-04-16T05:35:22.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:21 smithi027 ceph-mon[29206]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi027.front.sepia.ceph.com:3000"}]: dispatch 2024-04-16T05:35:22.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:22.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T05:35:22.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:21 smithi027 ceph-mon[29206]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T05:35:22.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi027.front.sepia.ceph.com:9095"}]: dispatch 2024-04-16T05:35:22.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:21 smithi027 ceph-mon[29206]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi027.front.sepia.ceph.com:9095"}]: dispatch 2024-04-16T05:35:22.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:22.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:35:22.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:21 smithi027 ceph-mon[29206]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:24.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:24 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:24.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:24 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:24.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:24 smithi027 ceph-mon[29206]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:24.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:24 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:24.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:24 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:24.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:24.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:24.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:24 smithi122 ceph-mon[35998]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:24.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:24.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:26.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:26 smithi027 ceph-mon[29206]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:27.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:26 smithi122 ceph-mon[35998]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:27.751 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:27 smithi027 ceph-mon[29206]: from='osd.0 [v2:172.21.15.27:6802/1557847192,v1:172.21.15.27:6803/1557847192]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-16T05:35:28.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:27 smithi122 ceph-mon[35998]: from='osd.0 [v2:172.21.15.27:6802/1557847192,v1:172.21.15.27:6803/1557847192]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-16T05:35:28.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:28 smithi027 ceph-mon[29206]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:28.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:28 smithi027 ceph-mon[29206]: from='osd.0 [v2:172.21.15.27:6802/1557847192,v1:172.21.15.27:6803/1557847192]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-16T05:35:28.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:28 smithi027 ceph-mon[29206]: osdmap e7: 1 total, 0 up, 1 in 2024-04-16T05:35:28.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:28 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T05:35:28.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:28 smithi027 ceph-mon[29206]: from='osd.0 [v2:172.21.15.27:6802/1557847192,v1:172.21.15.27:6803/1557847192]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-16T05:35:29.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:28 smithi122 ceph-mon[35998]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:29.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:28 smithi122 ceph-mon[35998]: from='osd.0 [v2:172.21.15.27:6802/1557847192,v1:172.21.15.27:6803/1557847192]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-16T05:35:29.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:28 smithi122 ceph-mon[35998]: osdmap e7: 1 total, 0 up, 1 in 2024-04-16T05:35:29.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:28 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T05:35:29.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:28 smithi122 ceph-mon[35998]: from='osd.0 [v2:172.21.15.27:6802/1557847192,v1:172.21.15.27:6803/1557847192]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-16T05:35:29.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:29.578+0000 7f82ef400700 1 -- 172.21.15.27:0/3151041406 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f82f8104640 con 0x7f82dc069210 2024-04-16T05:35:29.578 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 0 on host 'smithi027' 2024-04-16T05:35:29.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:29.581+0000 7f82fe200700 1 -- 172.21.15.27:0/3151041406 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f82dc069210 msgr2=0x7f82dc06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:35:29.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:29.581+0000 7f82fe200700 1 --2- 172.21.15.27:0/3151041406 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f82dc069210 0x7f82dc06b6c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f82e8007fd0 tx=0x7f82e8009040 comp rx=0 tx=0).stop 2024-04-16T05:35:29.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:29.581+0000 7f82fe200700 1 -- 172.21.15.27:0/3151041406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f81002d0 msgr2=0x7f82f8110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:35:29.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:29.581+0000 7f82fe200700 1 --2- 172.21.15.27:0/3151041406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f81002d0 0x7f82f8110070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f82e400b3b0 tx=0x7f82e4009e30 comp rx=0 tx=0).stop 2024-04-16T05:35:29.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:29.581+0000 7f82fe200700 1 -- 172.21.15.27:0/3151041406 shutdown_connections 2024-04-16T05:35:29.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:29.581+0000 7f82fe200700 1 --2- 172.21.15.27:0/3151041406 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f82dc069210 0x7f82dc06b6c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:35:29.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:29.581+0000 7f82fe200700 1 --2- 172.21.15.27:0/3151041406 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82f80ff020 0x7f82f810fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:35:29.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:29.581+0000 7f82fe200700 1 --2- 172.21.15.27:0/3151041406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82f81002d0 0x7f82f8110070 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:35:29.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:29.581+0000 7f82fe200700 1 -- 172.21.15.27:0/3151041406 >> 172.21.15.27:0/3151041406 conn(0x7f82f80fa950 msgr2=0x7f82f80fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:35:29.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:29.582+0000 7f82fe200700 1 -- 172.21.15.27:0/3151041406 shutdown_connections 2024-04-16T05:35:29.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:29.582+0000 7f82fe200700 1 -- 172.21.15.27:0/3151041406 wait complete. 2024-04-16T05:35:29.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:29 smithi027 ceph-mon[29206]: from='osd.0 [v2:172.21.15.27:6802/1557847192,v1:172.21.15.27:6803/1557847192]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-16T05:35:29.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:29 smithi027 ceph-mon[29206]: osdmap e8: 1 total, 0 up, 1 in 2024-04-16T05:35:29.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T05:35:29.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T05:35:29.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:29.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:29.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:29.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:30.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:29 smithi122 ceph-mon[35998]: from='osd.0 [v2:172.21.15.27:6802/1557847192,v1:172.21.15.27:6803/1557847192]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-16T05:35:30.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:29 smithi122 ceph-mon[35998]: osdmap e8: 1 total, 0 up, 1 in 2024-04-16T05:35:30.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T05:35:30.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T05:35:30.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:30.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:30.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:30.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:30.380 DEBUG:teuthology.orchestra.run.smithi027:osd.0> sudo journalctl -f -n 0 -u ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.0.service 2024-04-16T05:35:30.387 INFO:tasks.cephadm:Deploying osd.1 on smithi027 with /dev/vg_nvme/lv_3... 2024-04-16T05:35:30.387 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-16T05:35:30.897 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:30 smithi027 ceph-mon[29206]: purged_snaps scrub starts 2024-04-16T05:35:30.897 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:30 smithi027 ceph-mon[29206]: purged_snaps scrub ok 2024-04-16T05:35:30.897 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:30 smithi027 ceph-mon[29206]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:30.897 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:30 smithi027 ceph-mon[29206]: osd.0 [v2:172.21.15.27:6802/1557847192,v1:172.21.15.27:6803/1557847192] boot 2024-04-16T05:35:30.897 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:30 smithi027 ceph-mon[29206]: osdmap e9: 1 total, 1 up, 1 in 2024-04-16T05:35:30.897 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:30 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T05:35:31.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:30 smithi122 ceph-mon[35998]: purged_snaps scrub starts 2024-04-16T05:35:31.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:30 smithi122 ceph-mon[35998]: purged_snaps scrub ok 2024-04-16T05:35:31.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:30 smithi122 ceph-mon[35998]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-16T05:35:31.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:30 smithi122 ceph-mon[35998]: osd.0 [v2:172.21.15.27:6802/1557847192,v1:172.21.15.27:6803/1557847192] boot 2024-04-16T05:35:31.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:30 smithi122 ceph-mon[35998]: osdmap e9: 1 total, 1 up, 1 in 2024-04-16T05:35:31.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:30 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T05:35:32.070 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:35:32.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:32 smithi027 ceph-mon[29206]: osdmap e10: 1 total, 1 up, 1 in 2024-04-16T05:35:32.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:32 smithi027 ceph-mon[29206]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:32.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:32 smithi122 ceph-mon[35998]: osdmap e10: 1 total, 1 up, 1 in 2024-04-16T05:35:32.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:32 smithi122 ceph-mon[35998]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:34.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:34 smithi027 ceph-mon[29206]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:34.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:34 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:35:35.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:34 smithi122 ceph-mon[35998]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:35.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:34 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:35:35.435 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-16T05:35:35.435 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-16T05:35:35.435 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-16T05:35:35.435 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-16T05:35:35.436 INFO:teuthology.orchestra.run.smithi027.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0337119 s, 311 MB/s 2024-04-16T05:35:35.436 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-16T05:35:36.473 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_3 2024-04-16T05:35:37.003 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:35:37.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:36 smithi122 ceph-mon[35998]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:37.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:36 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:37.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:36 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:37.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:36 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:35:37.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:36 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:37.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:36 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:35:37.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:36 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:35:37.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:36 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:37.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:36 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:35:37.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:36 smithi027 ceph-mon[29206]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:37.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:36 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:37.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:36 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:37.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:36 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:35:37.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:36 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:37.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:36 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:35:37.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:36 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:35:37.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:36 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:37.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:36 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:35:37.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:37 smithi027 ceph-mon[29206]: Detected new or changed devices on smithi027 2024-04-16T05:35:37.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:37 smithi027 ceph-mon[29206]: Adjusting osd_memory_target on smithi027 to 12865M 2024-04-16T05:35:38.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:37 smithi122 ceph-mon[35998]: Detected new or changed devices on smithi027 2024-04-16T05:35:38.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:37 smithi122 ceph-mon[35998]: Adjusting osd_memory_target on smithi027 to 12865M 2024-04-16T05:35:38.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:38 smithi122 ceph-mon[35998]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:38.872 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:38 smithi027 ceph-mon[29206]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:39.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.750+0000 7fb48f400700 1 -- 172.21.15.27:0/1639062161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4880fe0e0 msgr2=0x7fb4880fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:35:39.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.750+0000 7fb48f400700 1 --2- 172.21.15.27:0/1639062161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4880fe0e0 0x7fb4880fe550 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb474004940 tx=0x7fb474009e30 comp rx=0 tx=0).stop 2024-04-16T05:35:39.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.750+0000 7fb48f400700 1 -- 172.21.15.27:0/1639062161 shutdown_connections 2024-04-16T05:35:39.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.751+0000 7fb48f400700 1 --2- 172.21.15.27:0/1639062161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4880fe0e0 0x7fb4880fe550 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:35:39.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.751+0000 7fb48f400700 1 --2- 172.21.15.27:0/1639062161 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4880fcef0 0x7fb4880fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:35:39.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.751+0000 7fb48f400700 1 -- 172.21.15.27:0/1639062161 >> 172.21.15.27:0/1639062161 conn(0x7fb4880f8530 msgr2=0x7fb4880fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:35:39.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.751+0000 7fb48f400700 1 -- 172.21.15.27:0/1639062161 shutdown_connections 2024-04-16T05:35:39.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.751+0000 7fb48f400700 1 -- 172.21.15.27:0/1639062161 wait complete. 2024-04-16T05:35:39.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.752+0000 7fb48f400700 1 Processor -- start 2024-04-16T05:35:39.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.752+0000 7fb48f400700 1 -- start start 2024-04-16T05:35:39.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.752+0000 7fb48f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4880fcef0 0x7fb48810fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:35:39.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.752+0000 7fb48f400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4880fe0e0 0x7fb488110180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:35:39.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.752+0000 7fb48f400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb488110780 con 0x7fb4880fcef0 2024-04-16T05:35:39.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.752+0000 7fb48f400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb488110fd0 con 0x7fb4880fe0e0 2024-04-16T05:35:39.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.753+0000 7fb48ca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4880fcef0 0x7fb48810fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:35:39.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.753+0000 7fb48ca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4880fcef0 0x7fb48810fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50308/0 (socket says 172.21.15.27:50308) 2024-04-16T05:35:39.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.753+0000 7fb48ca00700 1 -- 172.21.15.27:0/1988596905 learned_addr learned my addr 172.21.15.27:0/1988596905 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:35:39.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.753+0000 7fb487e00700 1 --2- 172.21.15.27:0/1988596905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4880fe0e0 0x7fb488110180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:35:39.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.753+0000 7fb48ca00700 1 -- 172.21.15.27:0/1988596905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4880fe0e0 msgr2=0x7fb488110180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:35:39.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.753+0000 7fb48ca00700 1 --2- 172.21.15.27:0/1988596905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4880fe0e0 0x7fb488110180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:35:39.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.753+0000 7fb48ca00700 1 -- 172.21.15.27:0/1988596905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb474005040 con 0x7fb4880fcef0 2024-04-16T05:35:39.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.754+0000 7fb48ca00700 1 --2- 172.21.15.27:0/1988596905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4880fcef0 0x7fb48810fc40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb47c002a80 tx=0x7fb47c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:35:39.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.754+0000 7fb484c00700 1 -- 172.21.15.27:0/1988596905 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb47c007cc0 con 0x7fb4880fcef0 2024-04-16T05:35:39.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.754+0000 7fb484c00700 1 -- 172.21.15.27:0/1988596905 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb47c00f040 con 0x7fb4880fcef0 2024-04-16T05:35:39.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.754+0000 7fb484c00700 1 -- 172.21.15.27:0/1988596905 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb47c00dad0 con 0x7fb4880fcef0 2024-04-16T05:35:39.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.754+0000 7fb48f400700 1 -- 172.21.15.27:0/1988596905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4881112b0 con 0x7fb4880fcef0 2024-04-16T05:35:39.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.754+0000 7fb48f400700 1 -- 172.21.15.27:0/1988596905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4881117f0 con 0x7fb4880fcef0 2024-04-16T05:35:39.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.756+0000 7fb48f400700 1 -- 172.21.15.27:0/1988596905 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb488101a70 con 0x7fb4880fcef0 2024-04-16T05:35:39.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.756+0000 7fb484c00700 1 -- 172.21.15.27:0/1988596905 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86573+0+0 (secure 0 0 0) 0x7fb47c00dc30 con 0x7fb4880fcef0 2024-04-16T05:35:39.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.757+0000 7fb484c00700 1 --2- 172.21.15.27:0/1988596905 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb470069210 0x7fb47006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:35:39.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.757+0000 7fb484c00700 1 -- 172.21.15.27:0/1988596905 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7fb47c086060 con 0x7fb4880fcef0 2024-04-16T05:35:39.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.758+0000 7fb487e00700 1 --2- 172.21.15.27:0/1988596905 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb470069210 0x7fb47006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:35:39.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.758+0000 7fb487e00700 1 --2- 172.21.15.27:0/1988596905 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb470069210 0x7fb47006b6c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb47400b3b0 tx=0x7fb474009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:35:39.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.762+0000 7fb484c00700 1 -- 172.21.15.27:0/1988596905 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb47c056c00 con 0x7fb4880fcef0 2024-04-16T05:35:39.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:35:39.999+0000 7fb48f400700 1 -- 172.21.15.27:0/1988596905 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fb488111a60 con 0x7fb470069210 2024-04-16T05:35:40.287 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:40 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:40.287 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:40 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:40.287 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:40 smithi027 ceph-mon[29206]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:40.287 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:40 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T05:35:40.287 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:40 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T05:35:40.287 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:40 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:35:40.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:40 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:40.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:40 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:40.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:40 smithi122 ceph-mon[35998]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:40.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:40 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T05:35:40.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:40 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T05:35:40.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:40 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:35:41.249 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:41 smithi027 ceph-mon[29206]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:35:41.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:41 smithi122 ceph-mon[35998]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:35:42.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:42 smithi122 ceph-mon[35998]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:42.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:42 smithi027 ceph-mon[29206]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:45.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:44 smithi122 ceph-mon[35998]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:45.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:44 smithi027 ceph-mon[29206]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:46.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:45 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:46.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:45 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:46.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:45 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:35:46.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:45 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:35:46.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:45 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:46.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:45 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/1251873235' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abec61aa-7a28-473b-b9d2-04e8d7de8a67"}]: dispatch 2024-04-16T05:35:46.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:45 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/1251873235' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "abec61aa-7a28-473b-b9d2-04e8d7de8a67"}]': finished 2024-04-16T05:35:46.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:45 smithi122 ceph-mon[35998]: osdmap e11: 2 total, 1 up, 2 in 2024-04-16T05:35:46.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:45 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T05:35:46.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:45 smithi122 ceph-mon[35998]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:46.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:45 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:46.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:45 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:46.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:45 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:35:46.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:45 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:35:46.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:45 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:35:46.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:45 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1251873235' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abec61aa-7a28-473b-b9d2-04e8d7de8a67"}]: dispatch 2024-04-16T05:35:46.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:45 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1251873235' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "abec61aa-7a28-473b-b9d2-04e8d7de8a67"}]': finished 2024-04-16T05:35:46.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:45 smithi027 ceph-mon[29206]: osdmap e11: 2 total, 1 up, 2 in 2024-04-16T05:35:46.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:45 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T05:35:46.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:45 smithi027 ceph-mon[29206]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:47.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:46 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/3219187669' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T05:35:47.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:46 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/3219187669' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T05:35:48.173 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:47 smithi122 ceph-mon[35998]: pgmap v58: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:48.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:47 smithi027 ceph-mon[29206]: pgmap v58: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:49.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:49 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:35:50.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:49 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:35:51.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:50 smithi122 ceph-mon[35998]: pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:51.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:50 smithi027 ceph-mon[29206]: pgmap v59: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:52.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:52 smithi027 ceph-mon[29206]: pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:53.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:52 smithi122 ceph-mon[35998]: pgmap v60: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:55.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:54 smithi122 ceph-mon[35998]: pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:55.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:54 smithi027 ceph-mon[29206]: pgmap v61: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:56.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:55 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T05:35:56.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:55 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:35:56.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:55 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T05:35:56.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:55 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:35:56.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:56 smithi027 ceph-mon[29206]: Deploying daemon osd.1 on smithi027 2024-04-16T05:35:56.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:56 smithi027 ceph-mon[29206]: pgmap v62: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:57.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:56 smithi122 ceph-mon[35998]: Deploying daemon osd.1 on smithi027 2024-04-16T05:35:57.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:56 smithi122 ceph-mon[35998]: pgmap v62: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:58.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:35:58 smithi027 ceph-mon[29206]: pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:35:59.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:35:58 smithi122 ceph-mon[35998]: pgmap v63: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:36:00.999 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:00 smithi027 ceph-mon[29206]: pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:36:01.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:00 smithi122 ceph-mon[35998]: pgmap v64: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:36:02.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:02 smithi027 ceph-mon[29206]: pgmap v65: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:36:03.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:02 smithi122 ceph-mon[35998]: pgmap v65: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:36:04.961 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:04 smithi122 ceph-mon[35998]: pgmap v66: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:36:04.962 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:04 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:04.962 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:04 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:04.962 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:04 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:36:04.962 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:04 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:36:04.981 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:04 smithi027 ceph-mon[29206]: pgmap v66: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:36:04.981 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:04 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:04.981 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:04 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:04.981 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:04 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:36:04.981 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:04 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:36:06.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:06 smithi027 ceph-mon[29206]: pgmap v67: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:36:06.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:06 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:06.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:06 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:07.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:06 smithi122 ceph-mon[35998]: pgmap v67: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:36:07.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:06 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:07.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:06 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:09.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:08 smithi122 ceph-mon[35998]: pgmap v68: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:36:09.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:08 smithi122 ceph-mon[35998]: from='osd.1 [v2:172.21.15.27:6810/2143934716,v1:172.21.15.27:6811/2143934716]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-16T05:36:09.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:08 smithi027 ceph-mon[29206]: pgmap v68: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:36:09.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:08 smithi027 ceph-mon[29206]: from='osd.1 [v2:172.21.15.27:6810/2143934716,v1:172.21.15.27:6811/2143934716]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-16T05:36:09.255 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 1 on host 'smithi027' 2024-04-16T05:36:09.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:09.255+0000 7fb484c00700 1 -- 172.21.15.27:0/1988596905 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb488111a60 con 0x7fb470069210 2024-04-16T05:36:09.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:09.258+0000 7fb48f400700 1 -- 172.21.15.27:0/1988596905 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb470069210 msgr2=0x7fb47006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:36:09.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:09.258+0000 7fb48f400700 1 --2- 172.21.15.27:0/1988596905 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb470069210 0x7fb47006b6c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb47400b3b0 tx=0x7fb474009e30 comp rx=0 tx=0).stop 2024-04-16T05:36:09.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:09.259+0000 7fb48f400700 1 -- 172.21.15.27:0/1988596905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4880fcef0 msgr2=0x7fb48810fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:36:09.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:09.259+0000 7fb48f400700 1 --2- 172.21.15.27:0/1988596905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4880fcef0 0x7fb48810fc40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb47c002a80 tx=0x7fb47c004600 comp rx=0 tx=0).stop 2024-04-16T05:36:09.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:09.259+0000 7fb48f400700 1 -- 172.21.15.27:0/1988596905 shutdown_connections 2024-04-16T05:36:09.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:09.259+0000 7fb48f400700 1 --2- 172.21.15.27:0/1988596905 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb470069210 0x7fb47006b6c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:36:09.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:09.259+0000 7fb48f400700 1 --2- 172.21.15.27:0/1988596905 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb4880fcef0 0x7fb48810fc40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:36:09.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:09.259+0000 7fb48f400700 1 --2- 172.21.15.27:0/1988596905 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb4880fe0e0 0x7fb488110180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:36:09.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:09.259+0000 7fb48f400700 1 -- 172.21.15.27:0/1988596905 >> 172.21.15.27:0/1988596905 conn(0x7fb4880f8530 msgr2=0x7fb4880f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:36:09.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:09.259+0000 7fb48f400700 1 -- 172.21.15.27:0/1988596905 shutdown_connections 2024-04-16T05:36:09.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:09.259+0000 7fb48f400700 1 -- 172.21.15.27:0/1988596905 wait complete. 2024-04-16T05:36:09.922 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:09 smithi027 ceph-mon[29206]: from='osd.1 [v2:172.21.15.27:6810/2143934716,v1:172.21.15.27:6811/2143934716]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-16T05:36:09.922 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:09 smithi027 ceph-mon[29206]: osdmap e12: 2 total, 1 up, 2 in 2024-04-16T05:36:09.923 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:09 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T05:36:09.923 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:09 smithi027 ceph-mon[29206]: from='osd.1 [v2:172.21.15.27:6810/2143934716,v1:172.21.15.27:6811/2143934716]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-16T05:36:09.923 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:09 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:09.923 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:09 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:10.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:09 smithi122 ceph-mon[35998]: from='osd.1 [v2:172.21.15.27:6810/2143934716,v1:172.21.15.27:6811/2143934716]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-16T05:36:10.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:09 smithi122 ceph-mon[35998]: osdmap e12: 2 total, 1 up, 2 in 2024-04-16T05:36:10.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:09 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T05:36:10.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:09 smithi122 ceph-mon[35998]: from='osd.1 [v2:172.21.15.27:6810/2143934716,v1:172.21.15.27:6811/2143934716]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-16T05:36:10.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:09 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:10.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:09 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:10.401 DEBUG:teuthology.orchestra.run.smithi027:osd.1> sudo journalctl -f -n 0 -u ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.1.service 2024-04-16T05:36:10.408 INFO:tasks.cephadm:Deploying osd.2 on smithi027 with /dev/vg_nvme/lv_2... 2024-04-16T05:36:10.408 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-16T05:36:10.946 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:36:11.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:10 smithi122 ceph-mon[35998]: pgmap v70: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:36:11.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:10 smithi122 ceph-mon[35998]: from='osd.1 [v2:172.21.15.27:6810/2143934716,v1:172.21.15.27:6811/2143934716]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-16T05:36:11.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:10 smithi122 ceph-mon[35998]: osdmap e13: 2 total, 1 up, 2 in 2024-04-16T05:36:11.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:10 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T05:36:11.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:10 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T05:36:11.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:10 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T05:36:11.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:10 smithi027 ceph-mon[29206]: pgmap v70: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:36:11.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:10 smithi027 ceph-mon[29206]: from='osd.1 [v2:172.21.15.27:6810/2143934716,v1:172.21.15.27:6811/2143934716]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-16T05:36:11.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:10 smithi027 ceph-mon[29206]: osdmap e13: 2 total, 1 up, 2 in 2024-04-16T05:36:11.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:10 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T05:36:11.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:10 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T05:36:11.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:10 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T05:36:12.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:11 smithi122 ceph-mon[35998]: purged_snaps scrub starts 2024-04-16T05:36:12.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:11 smithi122 ceph-mon[35998]: purged_snaps scrub ok 2024-04-16T05:36:12.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:11 smithi122 ceph-mon[35998]: osd.1 [v2:172.21.15.27:6810/2143934716,v1:172.21.15.27:6811/2143934716] boot 2024-04-16T05:36:12.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:11 smithi122 ceph-mon[35998]: osdmap e14: 2 total, 2 up, 2 in 2024-04-16T05:36:12.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:11 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T05:36:12.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:11 smithi122 ceph-mon[35998]: pgmap v73: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:36:12.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:11 smithi027 ceph-mon[29206]: purged_snaps scrub starts 2024-04-16T05:36:12.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:11 smithi027 ceph-mon[29206]: purged_snaps scrub ok 2024-04-16T05:36:12.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:11 smithi027 ceph-mon[29206]: osd.1 [v2:172.21.15.27:6810/2143934716,v1:172.21.15.27:6811/2143934716] boot 2024-04-16T05:36:12.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:11 smithi027 ceph-mon[29206]: osdmap e14: 2 total, 2 up, 2 in 2024-04-16T05:36:12.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:11 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T05:36:12.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:11 smithi027 ceph-mon[29206]: pgmap v73: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-04-16T05:36:13.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:12 smithi122 ceph-mon[35998]: osdmap e15: 2 total, 2 up, 2 in 2024-04-16T05:36:13.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:12 smithi027 ceph-mon[29206]: osdmap e15: 2 total, 2 up, 2 in 2024-04-16T05:36:14.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:13 smithi122 ceph-mon[35998]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:14.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:13 smithi027 ceph-mon[29206]: pgmap v75: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:15.002 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-16T05:36:15.003 INFO:teuthology.orchestra.run.smithi027.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-16T05:36:15.003 INFO:teuthology.orchestra.run.smithi027.stderr: stderr: 10+0 records in 2024-04-16T05:36:15.003 INFO:teuthology.orchestra.run.smithi027.stderr:10+0 records out 2024-04-16T05:36:15.003 INFO:teuthology.orchestra.run.smithi027.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0347876 s, 301 MB/s 2024-04-16T05:36:15.003 INFO:teuthology.orchestra.run.smithi027.stderr:--> Zapping successful for: 2024-04-16T05:36:15.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:15 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:15.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:15 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:15.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:15 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:15.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:15 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:16.025 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi027:vg_nvme/lv_2 2024-04-16T05:36:16.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:16 smithi027 ceph-mon[29206]: pgmap v76: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:16.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:16 smithi122 ceph-mon[35998]: pgmap v76: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:17.522 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:36:18.926 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:18 smithi027 ceph-mon[29206]: pgmap v77: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:18.941 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:18 smithi122 ceph-mon[35998]: pgmap v77: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:19.757 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:19 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:36:20.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:19 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:36:20.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.443+0000 7f2811200700 1 -- 172.21.15.27:0/4022867996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f280c0708f0 msgr2=0x7f280c070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:36:20.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.443+0000 7f2811200700 1 --2- 172.21.15.27:0/4022867996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f280c0708f0 0x7f280c070d60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f27f8004940 tx=0x7f27f8009e30 comp rx=0 tx=0).stop 2024-04-16T05:36:20.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.444+0000 7f2811200700 1 -- 172.21.15.27:0/4022867996 shutdown_connections 2024-04-16T05:36:20.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.444+0000 7f2811200700 1 --2- 172.21.15.27:0/4022867996 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f280c0708f0 0x7f280c070d60 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:36:20.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.444+0000 7f2811200700 1 --2- 172.21.15.27:0/4022867996 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f280c071ea0 0x7f280c0703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:36:20.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.444+0000 7f2811200700 1 -- 172.21.15.27:0/4022867996 >> 172.21.15.27:0/4022867996 conn(0x7f280c0f83d0 msgr2=0x7f280c0fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:36:20.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.444+0000 7f2811200700 1 -- 172.21.15.27:0/4022867996 shutdown_connections 2024-04-16T05:36:20.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.444+0000 7f2811200700 1 -- 172.21.15.27:0/4022867996 wait complete. 2024-04-16T05:36:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.445+0000 7f2811200700 1 Processor -- start 2024-04-16T05:36:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.445+0000 7f2811200700 1 -- start start 2024-04-16T05:36:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.445+0000 7f2811200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f280c0708f0 0x7f280c10fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:36:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.445+0000 7f2811200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f280c071ea0 0x7f280c10ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:36:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.445+0000 7f2811200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280c1105d0 con 0x7f280c0708f0 2024-04-16T05:36:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.445+0000 7f2811200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f280c110e20 con 0x7f280c071ea0 2024-04-16T05:36:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.446+0000 7f280b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f280c0708f0 0x7f280c10fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:36:20.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.446+0000 7f280b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f280c0708f0 0x7f280c10fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39932/0 (socket says 172.21.15.27:39932) 2024-04-16T05:36:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.446+0000 7f280b400700 1 -- 172.21.15.27:0/2573552510 learned_addr learned my addr 172.21.15.27:0/2573552510 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:36:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.446+0000 7f280aa00700 1 --2- 172.21.15.27:0/2573552510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f280c071ea0 0x7f280c10ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:36:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.446+0000 7f280b400700 1 -- 172.21.15.27:0/2573552510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f280c071ea0 msgr2=0x7f280c10ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:36:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.446+0000 7f280b400700 1 --2- 172.21.15.27:0/2573552510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f280c071ea0 0x7f280c10ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:36:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.446+0000 7f280b400700 1 -- 172.21.15.27:0/2573552510 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f8005040 con 0x7f280c0708f0 2024-04-16T05:36:20.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.447+0000 7f280b400700 1 --2- 172.21.15.27:0/2573552510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f280c0708f0 0x7f280c10fa90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00c8e0 tx=0x7f27fc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:36:20.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.447+0000 7f2803400700 1 -- 172.21.15.27:0/2573552510 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27fc00a940 con 0x7f280c0708f0 2024-04-16T05:36:20.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.447+0000 7f2803400700 1 -- 172.21.15.27:0/2573552510 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27fc00ce80 con 0x7f280c0708f0 2024-04-16T05:36:20.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.447+0000 7f2803400700 1 -- 172.21.15.27:0/2573552510 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f27fc0113a0 con 0x7f280c0708f0 2024-04-16T05:36:20.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.447+0000 7f2811200700 1 -- 172.21.15.27:0/2573552510 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f280c111100 con 0x7f280c0708f0 2024-04-16T05:36:20.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.447+0000 7f2811200700 1 -- 172.21.15.27:0/2573552510 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f280c111640 con 0x7f280c0708f0 2024-04-16T05:36:20.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.448+0000 7f2811200700 1 -- 172.21.15.27:0/2573552510 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f280c05f5f0 con 0x7f280c0708f0 2024-04-16T05:36:20.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.450+0000 7f2803400700 1 -- 172.21.15.27:0/2573552510 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 18) v1 ==== 86573+0+0 (secure 0 0 0) 0x7f27fc012400 con 0x7f280c0708f0 2024-04-16T05:36:20.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.450+0000 7f2803400700 1 --2- 172.21.15.27:0/2573552510 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f27f0069210 0x7f27f006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:36:20.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.451+0000 7f2803400700 1 -- 172.21.15.27:0/2573552510 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7f27fc086570 con 0x7f280c0708f0 2024-04-16T05:36:20.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.451+0000 7f280aa00700 1 --2- 172.21.15.27:0/2573552510 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f27f0069210 0x7f27f006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:36:20.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.452+0000 7f280aa00700 1 --2- 172.21.15.27:0/2573552510 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f27f0069210 0x7f27f006b6c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f27f8004f90 tx=0x7f27f8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:36:20.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.456+0000 7f2803400700 1 -- 172.21.15.27:0/2573552510 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f27fc056fe0 con 0x7f280c0708f0 2024-04-16T05:36:20.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:20.686+0000 7f2811200700 1 -- 172.21.15.27:0/2573552510 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f280c1119b0 con 0x7f27f0069210 2024-04-16T05:36:20.962 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:20 smithi027 ceph-mon[29206]: pgmap v78: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:21.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:20 smithi122 ceph-mon[35998]: pgmap v78: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:21.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:21 smithi027 ceph-mon[29206]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:36:21.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T05:36:21.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T05:36:21.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:36:21.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:21.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:21.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:36:21.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:36:21.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:21.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:36:21.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:36:21.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:21.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:36:22.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:21 smithi122 ceph-mon[35998]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi027:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:36:22.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T05:36:22.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T05:36:22.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:36:22.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:22.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:22.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:36:22.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:36:22.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:22.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:36:22.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:36:22.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:22.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:36:22.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:22 smithi027 ceph-mon[29206]: Detected new or changed devices on smithi027 2024-04-16T05:36:22.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:22 smithi027 ceph-mon[29206]: Adjusting osd_memory_target on smithi027 to 6432M 2024-04-16T05:36:22.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:22 smithi027 ceph-mon[29206]: pgmap v79: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:23.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:22 smithi122 ceph-mon[35998]: Detected new or changed devices on smithi027 2024-04-16T05:36:23.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:22 smithi122 ceph-mon[35998]: Adjusting osd_memory_target on smithi027 to 6432M 2024-04-16T05:36:23.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:22 smithi122 ceph-mon[35998]: pgmap v79: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:24.873 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:24 smithi027 ceph-mon[29206]: pgmap v80: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:25.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:24 smithi122 ceph-mon[35998]: pgmap v80: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:25.958 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:25 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1004280737' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e50778e4-bf1f-45eb-aeab-05e8dad8fa89"}]: dispatch 2024-04-16T05:36:25.959 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:25 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1004280737' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e50778e4-bf1f-45eb-aeab-05e8dad8fa89"}]': finished 2024-04-16T05:36:25.959 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:25 smithi027 ceph-mon[29206]: osdmap e16: 3 total, 2 up, 3 in 2024-04-16T05:36:25.959 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:25 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T05:36:26.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:25 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/1004280737' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e50778e4-bf1f-45eb-aeab-05e8dad8fa89"}]: dispatch 2024-04-16T05:36:26.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:25 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/1004280737' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e50778e4-bf1f-45eb-aeab-05e8dad8fa89"}]': finished 2024-04-16T05:36:26.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:25 smithi122 ceph-mon[35998]: osdmap e16: 3 total, 2 up, 3 in 2024-04-16T05:36:26.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:25 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T05:36:26.939 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:26 smithi027 ceph-mon[29206]: pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:26.939 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:26 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/854204454' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T05:36:27.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:26 smithi122 ceph-mon[35998]: pgmap v82: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:27.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:26 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/854204454' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T05:36:29.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:28 smithi122 ceph-mon[35998]: pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:29.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:28 smithi027 ceph-mon[29206]: pgmap v83: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:29.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:29.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:29.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:36:29.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:36:29.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:29.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:29 smithi027 ceph-mon[29206]: pgmap v84: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:30.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:30.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:30.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:36:30.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:36:30.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:30.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:29 smithi122 ceph-mon[35998]: pgmap v84: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:32.895 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:32 smithi027 ceph-mon[29206]: pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:33.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:32 smithi122 ceph-mon[35998]: pgmap v85: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:34.748 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:34 smithi027 ceph-mon[29206]: pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:34.748 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:34 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:36:35.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:34 smithi122 ceph-mon[35998]: pgmap v86: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:35.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:34 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:36:35.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T05:36:35.839 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:36:36.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T05:36:36.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:36:37.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:36 smithi122 ceph-mon[35998]: Deploying daemon osd.2 on smithi027 2024-04-16T05:36:37.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:36 smithi122 ceph-mon[35998]: pgmap v87: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:37.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:36 smithi027 ceph-mon[29206]: Deploying daemon osd.2 on smithi027 2024-04-16T05:36:37.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:36 smithi027 ceph-mon[29206]: pgmap v87: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:39.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:38 smithi122 ceph-mon[35998]: pgmap v88: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:39.034 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:38 smithi027 ceph-mon[29206]: pgmap v88: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:40.853 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:40 smithi027 ceph-mon[29206]: pgmap v89: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:41.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:40 smithi122 ceph-mon[35998]: pgmap v89: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:42.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:42 smithi027 ceph-mon[29206]: pgmap v90: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:43.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:42 smithi122 ceph-mon[35998]: pgmap v90: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:44.879 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:44 smithi027 ceph-mon[29206]: pgmap v91: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:44.879 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:44 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:44.879 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:44 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:36:44.879 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:44 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:45.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:44 smithi122 ceph-mon[35998]: pgmap v91: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:45.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:44 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:45.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:44 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:36:45.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:44 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:47.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:46 smithi122 ceph-mon[35998]: pgmap v92: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:47.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:46 smithi027 ceph-mon[29206]: pgmap v92: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:48.981 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:48 smithi027 ceph-mon[29206]: pgmap v93: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:48.981 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:48 smithi027 ceph-mon[29206]: from='osd.2 [v2:172.21.15.27:6818/3370929967,v1:172.21.15.27:6819/3370929967]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-16T05:36:49.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:48 smithi122 ceph-mon[35998]: pgmap v93: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:49.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:48 smithi122 ceph-mon[35998]: from='osd.2 [v2:172.21.15.27:6818/3370929967,v1:172.21.15.27:6819/3370929967]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-16T05:36:49.083 INFO:teuthology.orchestra.run.smithi027.stdout:Created osd(s) 2 on host 'smithi027' 2024-04-16T05:36:49.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:49.083+0000 7f2803400700 1 -- 172.21.15.27:0/2573552510 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f280c1119b0 con 0x7f27f0069210 2024-04-16T05:36:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:49.086+0000 7f2811200700 1 -- 172.21.15.27:0/2573552510 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f27f0069210 msgr2=0x7f27f006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:36:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:49.086+0000 7f2811200700 1 --2- 172.21.15.27:0/2573552510 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f27f0069210 0x7f27f006b6c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f27f8004f90 tx=0x7f27f8009e30 comp rx=0 tx=0).stop 2024-04-16T05:36:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:49.086+0000 7f2811200700 1 -- 172.21.15.27:0/2573552510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f280c0708f0 msgr2=0x7f280c10fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:36:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:49.086+0000 7f2811200700 1 --2- 172.21.15.27:0/2573552510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f280c0708f0 0x7f280c10fa90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00c8e0 tx=0x7f27fc0044c0 comp rx=0 tx=0).stop 2024-04-16T05:36:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:49.086+0000 7f2811200700 1 -- 172.21.15.27:0/2573552510 shutdown_connections 2024-04-16T05:36:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:49.086+0000 7f2811200700 1 --2- 172.21.15.27:0/2573552510 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f27f0069210 0x7f27f006b6c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:36:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:49.086+0000 7f2811200700 1 --2- 172.21.15.27:0/2573552510 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f280c0708f0 0x7f280c10fa90 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:36:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:49.086+0000 7f2811200700 1 --2- 172.21.15.27:0/2573552510 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f280c071ea0 0x7f280c10ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:36:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:49.086+0000 7f2811200700 1 -- 172.21.15.27:0/2573552510 >> 172.21.15.27:0/2573552510 conn(0x7f280c0f83d0 msgr2=0x7f280c0f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:36:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:49.086+0000 7f2811200700 1 -- 172.21.15.27:0/2573552510 shutdown_connections 2024-04-16T05:36:49.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:36:49.087+0000 7f2811200700 1 -- 172.21.15.27:0/2573552510 wait complete. 2024-04-16T05:36:50.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:49 smithi122 ceph-mon[35998]: from='osd.2 [v2:172.21.15.27:6818/3370929967,v1:172.21.15.27:6819/3370929967]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-16T05:36:50.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:49 smithi122 ceph-mon[35998]: osdmap e17: 3 total, 2 up, 3 in 2024-04-16T05:36:50.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:49 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T05:36:50.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:49 smithi122 ceph-mon[35998]: from='osd.2 [v2:172.21.15.27:6818/3370929967,v1:172.21.15.27:6819/3370929967]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-16T05:36:50.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:49 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:50.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:49 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:50.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:49 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:36:50.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:49 smithi027 ceph-mon[29206]: from='osd.2 [v2:172.21.15.27:6818/3370929967,v1:172.21.15.27:6819/3370929967]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-16T05:36:50.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:49 smithi027 ceph-mon[29206]: osdmap e17: 3 total, 2 up, 3 in 2024-04-16T05:36:50.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:49 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T05:36:50.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:49 smithi027 ceph-mon[29206]: from='osd.2 [v2:172.21.15.27:6818/3370929967,v1:172.21.15.27:6819/3370929967]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-16T05:36:50.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:49 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:50.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:49 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:50.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:49 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:36:50.246 DEBUG:teuthology.orchestra.run.smithi027:osd.2> sudo journalctl -f -n 0 -u ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.2.service 2024-04-16T05:36:50.254 INFO:tasks.cephadm:Deploying osd.3 on smithi122 with /dev/vg_nvme/lv_4... 2024-04-16T05:36:50.254 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-16T05:36:50.754 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi122/config 2024-04-16T05:36:50.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:50 smithi027 ceph-mon[29206]: pgmap v95: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:50.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:50 smithi027 ceph-mon[29206]: from='osd.2 [v2:172.21.15.27:6818/3370929967,v1:172.21.15.27:6819/3370929967]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-16T05:36:50.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:50 smithi027 ceph-mon[29206]: osdmap e18: 3 total, 2 up, 3 in 2024-04-16T05:36:50.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:50 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T05:36:50.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:50 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T05:36:50.905 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:50 smithi122 ceph-mon[35998]: pgmap v95: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:50.905 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:50 smithi122 ceph-mon[35998]: from='osd.2 [v2:172.21.15.27:6818/3370929967,v1:172.21.15.27:6819/3370929967]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]': finished 2024-04-16T05:36:50.905 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:50 smithi122 ceph-mon[35998]: osdmap e18: 3 total, 2 up, 3 in 2024-04-16T05:36:50.906 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:50 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T05:36:50.906 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:50 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T05:36:52.655 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:52 smithi122 ceph-mon[35998]: purged_snaps scrub starts 2024-04-16T05:36:52.655 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:52 smithi122 ceph-mon[35998]: purged_snaps scrub ok 2024-04-16T05:36:52.655 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:52 smithi122 ceph-mon[35998]: osd.2 [v2:172.21.15.27:6818/3370929967,v1:172.21.15.27:6819/3370929967] boot 2024-04-16T05:36:52.655 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:52 smithi122 ceph-mon[35998]: osdmap e19: 3 total, 3 up, 3 in 2024-04-16T05:36:52.655 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:52 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T05:36:52.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:52 smithi027 ceph-mon[29206]: purged_snaps scrub starts 2024-04-16T05:36:52.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:52 smithi027 ceph-mon[29206]: purged_snaps scrub ok 2024-04-16T05:36:52.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:52 smithi027 ceph-mon[29206]: osd.2 [v2:172.21.15.27:6818/3370929967,v1:172.21.15.27:6819/3370929967] boot 2024-04-16T05:36:52.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:52 smithi027 ceph-mon[29206]: osdmap e19: 3 total, 3 up, 3 in 2024-04-16T05:36:52.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:52 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T05:36:53.533 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-16T05:36:53.534 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-16T05:36:53.534 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-04-16T05:36:53.534 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-04-16T05:36:53.534 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0331041 s, 317 MB/s 2024-04-16T05:36:53.534 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-04-16T05:36:53.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:53 smithi122 ceph-mon[35998]: pgmap v98: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:53.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:53 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-16T05:36:53.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:53 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-16T05:36:53.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:53 smithi122 ceph-mon[35998]: osdmap e20: 3 total, 3 up, 3 in 2024-04-16T05:36:53.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:53 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-16T05:36:53.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:53 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:53.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:53 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:53.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:53 smithi027 ceph-mon[29206]: pgmap v98: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-04-16T05:36:53.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:53 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-16T05:36:53.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:53 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-16T05:36:53.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:53 smithi027 ceph-mon[29206]: osdmap e20: 3 total, 3 up, 3 in 2024-04-16T05:36:53.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:53 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-16T05:36:53.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:53 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:53.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:53 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:54.270 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi122:vg_nvme/lv_4 2024-04-16T05:36:54.713 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi122/config 2024-04-16T05:36:54.721 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:54 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-16T05:36:54.721 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:54 smithi027 ceph-mon[29206]: osdmap e21: 3 total, 3 up, 3 in 2024-04-16T05:36:54.721 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:54 smithi027 ceph-mon[29206]: pgmap v101: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-16T05:36:54.722 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:36:54 smithi027 sudo[64429]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-16T05:36:54.722 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:36:54 smithi027 sudo[64429]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T05:36:54.722 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:36:54 smithi027 sudo[64429]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T05:36:54.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:54 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-16T05:36:54.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:54 smithi122 ceph-mon[35998]: osdmap e21: 3 total, 3 up, 3 in 2024-04-16T05:36:54.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:54 smithi122 ceph-mon[35998]: pgmap v101: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-16T05:36:55.057 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:36:54 smithi027 sudo[64429]: pam_unix(sudo:session): session closed for user root 2024-04-16T05:36:55.725 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:55 smithi027 ceph-mon[29206]: osdmap e22: 3 total, 3 up, 3 in 2024-04-16T05:36:55.726 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:36:55 smithi027 sudo[64480]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-16T05:36:55.726 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:36:55 smithi027 sudo[64480]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T05:36:55.726 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:36:55 smithi027 sudo[64480]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T05:36:55.726 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:36:55 smithi027 sudo[64480]: pam_unix(sudo:session): session closed for user root 2024-04-16T05:36:55.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:55 smithi122 ceph-mon[35998]: osdmap e22: 3 total, 3 up, 3 in 2024-04-16T05:36:56.556 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:36:56 smithi027 sudo[64489]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-16T05:36:56.556 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:36:56 smithi027 sudo[64489]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T05:36:56.556 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:36:56 smithi027 sudo[64489]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T05:36:56.556 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:36:56 smithi027 sudo[64489]: pam_unix(sudo:session): session closed for user root 2024-04-16T05:36:56.576 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.577+0000 7f9452a00700 1 -- 172.21.15.122:0/4239068127 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f944c0709a0 msgr2=0x7f944c070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:36:56.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.577+0000 7f9452a00700 1 --2- 172.21.15.122:0/4239068127 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f944c0709a0 0x7f944c070e10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f9434004940 tx=0x7f9434009e30 comp rx=0 tx=0).stop 2024-04-16T05:36:56.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.577+0000 7f9452a00700 1 -- 172.21.15.122:0/4239068127 shutdown_connections 2024-04-16T05:36:56.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.577+0000 7f9452a00700 1 --2- 172.21.15.122:0/4239068127 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f944c0709a0 0x7f944c070e10 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:36:56.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.577+0000 7f9452a00700 1 --2- 172.21.15.122:0/4239068127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f944c071f50 0x7f944c070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:36:56.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.577+0000 7f9452a00700 1 -- 172.21.15.122:0/4239068127 >> 172.21.15.122:0/4239068127 conn(0x7f944c0f8530 msgr2=0x7f944c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:36:56.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.577+0000 7f9452a00700 1 -- 172.21.15.122:0/4239068127 shutdown_connections 2024-04-16T05:36:56.577 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.577+0000 7f9452a00700 1 -- 172.21.15.122:0/4239068127 wait complete. 2024-04-16T05:36:56.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.578+0000 7f9452a00700 1 Processor -- start 2024-04-16T05:36:56.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.578+0000 7f9452a00700 1 -- start start 2024-04-16T05:36:56.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.579+0000 7f9452a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f944c0709a0 0x7f944c10fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:36:56.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.579+0000 7f9452a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f944c071f50 0x7f944c110270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:36:56.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.579+0000 7f9452a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944c110870 con 0x7f944c071f50 2024-04-16T05:36:56.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.579+0000 7f9452a00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944c110fb0 con 0x7f944c0709a0 2024-04-16T05:36:56.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.579+0000 7f944be00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f944c0709a0 0x7f944c10fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:36:56.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.579+0000 7f944be00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f944c0709a0 0x7f944c10fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:40626/0 (socket says 172.21.15.122:40626) 2024-04-16T05:36:56.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.579+0000 7f944be00700 1 -- 172.21.15.122:0/498774436 learned_addr learned my addr 172.21.15.122:0/498774436 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:36:56.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.579+0000 7f9443400700 1 --2- 172.21.15.122:0/498774436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f944c071f50 0x7f944c110270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:36:56.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.579+0000 7f944be00700 1 -- 172.21.15.122:0/498774436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f944c071f50 msgr2=0x7f944c110270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:36:56.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.579+0000 7f944be00700 1 --2- 172.21.15.122:0/498774436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f944c071f50 0x7f944c110270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:36:56.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.579+0000 7f944be00700 1 -- 172.21.15.122:0/498774436 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9434005040 con 0x7f944c0709a0 2024-04-16T05:36:56.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.580+0000 7f944be00700 1 --2- 172.21.15.122:0/498774436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f944c0709a0 0x7f944c10fd30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f943c00cb10 tx=0x7f943c00ced0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:36:56.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.580+0000 7f9448c00700 1 -- 172.21.15.122:0/498774436 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f943c00a980 con 0x7f944c0709a0 2024-04-16T05:36:56.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.580+0000 7f9448c00700 1 -- 172.21.15.122:0/498774436 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f943c012420 con 0x7f944c0709a0 2024-04-16T05:36:56.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.580+0000 7f9452a00700 1 -- 172.21.15.122:0/498774436 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f944c111290 con 0x7f944c0709a0 2024-04-16T05:36:56.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.580+0000 7f9448c00700 1 -- 172.21.15.122:0/498774436 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f943c0115c0 con 0x7f944c0709a0 2024-04-16T05:36:56.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.580+0000 7f9452a00700 1 -- 172.21.15.122:0/498774436 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f944c111850 con 0x7f944c0709a0 2024-04-16T05:36:56.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.581+0000 7f9452a00700 1 -- 172.21.15.122:0/498774436 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f944c05f5f0 con 0x7f944c0709a0 2024-04-16T05:36:56.581 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.582+0000 7f9448c00700 1 -- 172.21.15.122:0/498774436 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f943c003020 con 0x7f944c0709a0 2024-04-16T05:36:56.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.583+0000 7f9448c00700 1 --2- 172.21.15.122:0/498774436 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f94300692b0 0x7f943006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:36:56.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.583+0000 7f9448c00700 1 -- 172.21.15.122:0/498774436 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3228+0+0 (secure 0 0 0) 0x7f943c086dc0 con 0x7f944c0709a0 2024-04-16T05:36:56.582 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.584+0000 7f9443400700 1 --2- 172.21.15.122:0/498774436 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f94300692b0 0x7f943006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:36:56.583 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.585+0000 7f9443400700 1 --2- 172.21.15.122:0/498774436 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f94300692b0 0x7f943006b760 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f943400b3b0 tx=0x7f9434009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:36:56.587 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.589+0000 7f9448c00700 1 -- 172.21.15.122:0/498774436 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f943c08a030 con 0x7f944c0709a0 2024-04-16T05:36:56.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:56 smithi122 ceph-mon[35998]: pgmap v103: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-16T05:36:56.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:56 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:56.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:56 smithi122 ceph-mon[35998]: mgrmap e19: smithi027.qgzypi(active, since 3m), standbys: smithi122.sgokkq 2024-04-16T05:36:56.806 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:36:56.807+0000 7f9452a00700 1 -- 172.21.15.122:0/498774436 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f944c111be0 con 0x7f94300692b0 2024-04-16T05:36:56.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:56 smithi027 ceph-mon[29206]: pgmap v103: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-04-16T05:36:56.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:56 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:36:56.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:56 smithi027 ceph-mon[29206]: mgrmap e19: smithi027.qgzypi(active, since 3m), standbys: smithi122.sgokkq 2024-04-16T05:36:57.557 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:36:57 smithi027 sudo[64520]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-16T05:36:57.557 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:36:57 smithi027 sudo[64520]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T05:36:57.557 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:36:57 smithi027 sudo[64520]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T05:36:57.558 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:36:57 smithi027 sudo[64520]: pam_unix(sudo:session): session closed for user root 2024-04-16T05:36:57.855 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:57 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T05:36:57.855 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:57 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T05:36:57.855 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:57 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:36:57.981 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:57 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T05:36:57.982 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:57 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T05:36:57.982 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:57 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:36:58.307 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:36:57 smithi027 sudo[64550]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-16T05:36:58.308 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:36:57 smithi027 sudo[64550]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T05:36:58.308 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:36:57 smithi027 sudo[64550]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T05:36:58.308 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:36:58 smithi027 sudo[64550]: pam_unix(sudo:session): session closed for user root 2024-04-16T05:36:58.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:58 smithi122 ceph-mon[35998]: from='client.24129 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:36:58.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:36:58 smithi122 ceph-mon[35998]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T05:36:59.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:58 smithi027 ceph-mon[29206]: from='client.24129 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:36:59.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:58 smithi027 ceph-mon[29206]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T05:36:59.057 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:36:58 smithi027 sudo[67240]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-16T05:36:59.057 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:36:58 smithi027 sudo[67240]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T05:36:59.058 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:36:58 smithi027 sudo[67240]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T05:36:59.058 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:36:58 smithi027 sudo[67240]: pam_unix(sudo:session): session closed for user root 2024-04-16T05:36:59.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:59 smithi027 sudo[67248]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-16T05:36:59.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:59 smithi027 sudo[67248]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T05:36:59.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:59 smithi027 sudo[67248]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T05:36:59.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:36:59 smithi027 sudo[67248]: pam_unix(sudo:session): session closed for user root 2024-04-16T05:37:00.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:00 smithi027 sudo[67251]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-16T05:37:00.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:00 smithi027 sudo[67251]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T05:37:00.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:00 smithi027 sudo[67251]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T05:37:00.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:00 smithi027 sudo[67251]: pam_unix(sudo:session): session closed for user root 2024-04-16T05:37:01.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:00 smithi122 sudo[39461]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-16T05:37:01.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:00 smithi122 sudo[39461]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T05:37:01.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:00 smithi122 sudo[39461]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T05:37:01.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:00 smithi122 sudo[39461]: pam_unix(sudo:session): session closed for user root 2024-04-16T05:37:01.860 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 sudo[39464]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-16T05:37:01.860 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 sudo[39464]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-16T05:37:01.861 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 sudo[39464]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-16T05:37:01.861 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 sudo[39464]: pam_unix(sudo:session): session closed for user root 2024-04-16T05:37:01.861 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 ceph-mon[35998]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-16T05:37:01.861 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 ceph-mon[35998]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T05:37:01.861 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 ceph-mon[35998]: Detected new or changed devices on smithi027 2024-04-16T05:37:01.861 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 ceph-mon[35998]: from='client.? 172.21.15.122:0/1406918988' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f55dd70-a9d7-4bc1-a79c-028e66571137"}]: dispatch 2024-04-16T05:37:01.861 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 ceph-mon[35998]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-16T05:37:01.861 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-16T05:37:01.861 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 ceph-mon[35998]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f55dd70-a9d7-4bc1-a79c-028e66571137"}]: dispatch 2024-04-16T05:37:01.861 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:37:01.861 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 ceph-mon[35998]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f55dd70-a9d7-4bc1-a79c-028e66571137"}]': finished 2024-04-16T05:37:01.861 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 ceph-mon[35998]: osdmap e23: 4 total, 3 up, 4 in 2024-04-16T05:37:01.861 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T05:37:01.861 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:01.861 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:01.862 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-16T05:37:01.862 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:37:01.862 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:37:01.862 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:37:01.862 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:01 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:37:02.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:01 smithi027 ceph-mon[29206]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-16T05:37:02.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:01 smithi027 ceph-mon[29206]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T05:37:02.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:01 smithi027 ceph-mon[29206]: Detected new or changed devices on smithi027 2024-04-16T05:37:02.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:01 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/1406918988' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f55dd70-a9d7-4bc1-a79c-028e66571137"}]: dispatch 2024-04-16T05:37:02.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:01 smithi027 ceph-mon[29206]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-16T05:37:02.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:01 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-16T05:37:02.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:01 smithi027 ceph-mon[29206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8f55dd70-a9d7-4bc1-a79c-028e66571137"}]: dispatch 2024-04-16T05:37:02.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:01 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:37:02.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:01 smithi027 ceph-mon[29206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8f55dd70-a9d7-4bc1-a79c-028e66571137"}]': finished 2024-04-16T05:37:02.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:01 smithi027 ceph-mon[29206]: osdmap e23: 4 total, 3 up, 4 in 2024-04-16T05:37:02.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:01 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T05:37:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:01 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:01 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:01 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-16T05:37:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:01 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:37:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:01 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:37:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:01 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:37:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:01 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:37:02.925 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:02 smithi027 ceph-mon[29206]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-16T05:37:02.925 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:02 smithi027 ceph-mon[29206]: Adjusting osd_memory_target on smithi027 to 4288M 2024-04-16T05:37:02.925 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:02 smithi027 ceph-mon[29206]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T05:37:02.925 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:02 smithi027 ceph-mon[29206]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-16T05:37:02.925 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:02 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:02.925 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:02 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:37:02.926 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:02 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:37:02.926 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:02 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:02.926 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:02 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:02.926 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:02 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:37:02.926 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:02 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/3324549161' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T05:37:03.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:02 smithi122 ceph-mon[35998]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-16T05:37:03.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:02 smithi122 ceph-mon[35998]: Adjusting osd_memory_target on smithi027 to 4288M 2024-04-16T05:37:03.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:02 smithi122 ceph-mon[35998]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T05:37:03.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:02 smithi122 ceph-mon[35998]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-16T05:37:03.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:02 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:03.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:02 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:37:03.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:02 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:37:03.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:02 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:03.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:02 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:03.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:02 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:37:03.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:02 smithi122 ceph-mon[35998]: from='client.? 172.21.15.122:0/3324549161' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T05:37:04.863 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:04 smithi027 ceph-mon[29206]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-16T05:37:04.863 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:04 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:37:05.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:04 smithi122 ceph-mon[35998]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-16T05:37:05.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:04 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:37:06.851 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:06 smithi122 ceph-mon[35998]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-16T05:37:06.935 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:06 smithi027 ceph-mon[29206]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-16T05:37:08.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:08 smithi122 ceph-mon[35998]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-16T05:37:08.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:08 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:08.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:08 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:08.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:08 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:37:08.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:08 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:37:08.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:08 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:09.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:08 smithi027 ceph-mon[29206]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-16T05:37:09.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:08 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:09.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:08 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:09.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:08 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:37:09.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:08 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:37:09.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:08 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:09.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:09 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T05:37:09.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:09 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:37:09.789 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:09 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T05:37:09.789 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:09 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:37:11.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:10 smithi122 ceph-mon[35998]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-16T05:37:11.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:10 smithi122 ceph-mon[35998]: Deploying daemon osd.3 on smithi122 2024-04-16T05:37:11.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:10 smithi027 ceph-mon[29206]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-16T05:37:11.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:10 smithi027 ceph-mon[29206]: Deploying daemon osd.3 on smithi122 2024-04-16T05:37:12.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:12 smithi122 ceph-mon[35998]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 5 op/s 2024-04-16T05:37:13.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:12 smithi027 ceph-mon[29206]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 5 op/s 2024-04-16T05:37:14.725 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:14 smithi122 ceph-mon[35998]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-16T05:37:15.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:14 smithi027 ceph-mon[29206]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-16T05:37:16.817 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:16 smithi122 ceph-mon[35998]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T05:37:17.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:16 smithi027 ceph-mon[29206]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T05:37:18.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:17 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:18.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:17 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:18.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:17 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:37:18.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:17 smithi122 ceph-mon[35998]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T05:37:18.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:17 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:18.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:17 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:18.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:17 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:37:18.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:17 smithi027 ceph-mon[29206]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T05:37:19.962 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:19 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:37:20.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:19 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:37:20.959 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:20 smithi122 ceph-mon[35998]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T05:37:21.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:20 smithi027 ceph-mon[29206]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T05:37:21.924 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:21.924 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:21.924 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:37:21.924 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:37:21.924 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:21.924 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:21 smithi122 ceph-mon[35998]: from='osd.3 [v2:172.21.15.122:6800/186542313,v1:172.21.15.122:6801/186542313]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T05:37:21.924 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:21 smithi122 ceph-mon[35998]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T05:37:21.924 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:21 smithi122 ceph-mon[35998]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T05:37:22.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:22.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:22.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:37:22.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:37:22.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:22.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:21 smithi027 ceph-mon[29206]: from='osd.3 [v2:172.21.15.122:6800/186542313,v1:172.21.15.122:6801/186542313]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T05:37:22.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:21 smithi027 ceph-mon[29206]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T05:37:22.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:21 smithi027 ceph-mon[29206]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T05:37:22.310 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 3 on host 'smithi122' 2024-04-16T05:37:22.310 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:22.310+0000 7f9448c00700 1 -- 172.21.15.122:0/498774436 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f944c111be0 con 0x7f94300692b0 2024-04-16T05:37:22.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:22.314+0000 7f9452a00700 1 -- 172.21.15.122:0/498774436 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f94300692b0 msgr2=0x7f943006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:37:22.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:22.314+0000 7f9452a00700 1 --2- 172.21.15.122:0/498774436 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f94300692b0 0x7f943006b760 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f943400b3b0 tx=0x7f9434009e30 comp rx=0 tx=0).stop 2024-04-16T05:37:22.313 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:22.314+0000 7f9452a00700 1 -- 172.21.15.122:0/498774436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f944c0709a0 msgr2=0x7f944c10fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:37:22.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:22.314+0000 7f9452a00700 1 --2- 172.21.15.122:0/498774436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f944c0709a0 0x7f944c10fd30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f943c00cb10 tx=0x7f943c00ced0 comp rx=0 tx=0).stop 2024-04-16T05:37:22.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:22.314+0000 7f9452a00700 1 -- 172.21.15.122:0/498774436 shutdown_connections 2024-04-16T05:37:22.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:22.314+0000 7f9452a00700 1 --2- 172.21.15.122:0/498774436 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f94300692b0 0x7f943006b760 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:37:22.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:22.314+0000 7f9452a00700 1 --2- 172.21.15.122:0/498774436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f944c0709a0 0x7f944c10fd30 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:37:22.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:22.314+0000 7f9452a00700 1 --2- 172.21.15.122:0/498774436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f944c071f50 0x7f944c110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:37:22.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:22.314+0000 7f9452a00700 1 -- 172.21.15.122:0/498774436 >> 172.21.15.122:0/498774436 conn(0x7f944c0f8530 msgr2=0x7f944c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:37:22.314 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:22.316+0000 7f9452a00700 1 -- 172.21.15.122:0/498774436 shutdown_connections 2024-04-16T05:37:22.315 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:22.316+0000 7f9452a00700 1 -- 172.21.15.122:0/498774436 wait complete. 2024-04-16T05:37:22.957 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:22 smithi122 ceph-mon[35998]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-16T05:37:22.957 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:22 smithi122 ceph-mon[35998]: osdmap e24: 4 total, 3 up, 4 in 2024-04-16T05:37:22.957 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:22 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T05:37:22.957 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:22 smithi122 ceph-mon[35998]: from='osd.3 [v2:172.21.15.122:6800/186542313,v1:172.21.15.122:6801/186542313]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:37:22.957 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:22 smithi122 ceph-mon[35998]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:37:22.958 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:22 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:22.958 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:22 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:22.958 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:22 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:37:23.018 DEBUG:teuthology.orchestra.run.smithi122:osd.3> sudo journalctl -f -n 0 -u ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.3.service 2024-04-16T05:37:23.024 INFO:tasks.cephadm:Deploying osd.4 on smithi122 with /dev/vg_nvme/lv_3... 2024-04-16T05:37:23.024 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-16T05:37:23.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:22 smithi027 ceph-mon[29206]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-16T05:37:23.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:22 smithi027 ceph-mon[29206]: osdmap e24: 4 total, 3 up, 4 in 2024-04-16T05:37:23.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:22 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T05:37:23.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:22 smithi027 ceph-mon[29206]: from='osd.3 [v2:172.21.15.122:6800/186542313,v1:172.21.15.122:6801/186542313]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:37:23.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:22 smithi027 ceph-mon[29206]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:37:23.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:22 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:23.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:22 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:23.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:22 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:37:24.230 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:23 smithi122 ceph-mon[35998]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-16T05:37:24.230 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:23 smithi122 ceph-mon[35998]: osdmap e25: 4 total, 3 up, 4 in 2024-04-16T05:37:24.230 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:23 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T05:37:24.231 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:23 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T05:37:24.231 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:23 smithi122 ceph-mon[35998]: osd.3 [v2:172.21.15.122:6800/186542313,v1:172.21.15.122:6801/186542313] boot 2024-04-16T05:37:24.231 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:23 smithi122 ceph-mon[35998]: osdmap e26: 4 total, 4 up, 4 in 2024-04-16T05:37:24.231 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:23 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T05:37:24.231 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:23 smithi122 ceph-mon[35998]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T05:37:24.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:23 smithi027 ceph-mon[29206]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-16T05:37:24.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:23 smithi027 ceph-mon[29206]: osdmap e25: 4 total, 3 up, 4 in 2024-04-16T05:37:24.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:23 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T05:37:24.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:23 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T05:37:24.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:23 smithi027 ceph-mon[29206]: osd.3 [v2:172.21.15.122:6800/186542313,v1:172.21.15.122:6801/186542313] boot 2024-04-16T05:37:24.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:23 smithi027 ceph-mon[29206]: osdmap e26: 4 total, 4 up, 4 in 2024-04-16T05:37:24.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:23 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T05:37:24.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:23 smithi027 ceph-mon[29206]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-04-16T05:37:24.382 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi122/config 2024-04-16T05:37:25.136 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:24 smithi122 ceph-mon[35998]: purged_snaps scrub starts 2024-04-16T05:37:25.136 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:24 smithi122 ceph-mon[35998]: purged_snaps scrub ok 2024-04-16T05:37:25.136 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:24 smithi122 ceph-mon[35998]: osdmap e27: 4 total, 4 up, 4 in 2024-04-16T05:37:25.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:24 smithi027 ceph-mon[29206]: purged_snaps scrub starts 2024-04-16T05:37:25.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:24 smithi027 ceph-mon[29206]: purged_snaps scrub ok 2024-04-16T05:37:25.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:24 smithi027 ceph-mon[29206]: osdmap e27: 4 total, 4 up, 4 in 2024-04-16T05:37:26.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:26 smithi027 ceph-mon[29206]: osdmap e28: 4 total, 4 up, 4 in 2024-04-16T05:37:26.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:26 smithi027 ceph-mon[29206]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:26.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:26 smithi122 ceph-mon[35998]: osdmap e28: 4 total, 4 up, 4 in 2024-04-16T05:37:26.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:26 smithi122 ceph-mon[35998]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:27.318 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-16T05:37:27.318 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-16T05:37:27.318 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-04-16T05:37:27.318 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-04-16T05:37:27.318 INFO:teuthology.orchestra.run.smithi122.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0331043 s, 317 MB/s 2024-04-16T05:37:27.318 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-04-16T05:37:27.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:27 smithi122 ceph-mon[35998]: osdmap e29: 4 total, 4 up, 4 in 2024-04-16T05:37:27.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:27 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:27.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:27 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:27.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:27 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:37:27.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:27 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:27.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:27 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:37:27.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:27 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:37:27.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:27 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:27.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:27 smithi027 ceph-mon[29206]: osdmap e29: 4 total, 4 up, 4 in 2024-04-16T05:37:27.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:27 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:27.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:27 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:27.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:27 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:37:27.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:27 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:27.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:27 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:37:27.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:27 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:37:27.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:27 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:28.117 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi122:vg_nvme/lv_3 2024-04-16T05:37:28.582 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi122/config 2024-04-16T05:37:28.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:28 smithi122 ceph-mon[35998]: Detected new or changed devices on smithi122 2024-04-16T05:37:28.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:28 smithi122 ceph-mon[35998]: Adjusting osd_memory_target on smithi122 to 15937M 2024-04-16T05:37:28.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:28 smithi122 ceph-mon[35998]: osdmap e30: 4 total, 4 up, 4 in 2024-04-16T05:37:28.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:28 smithi122 ceph-mon[35998]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-16T05:37:28.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:28 smithi027 ceph-mon[29206]: Detected new or changed devices on smithi122 2024-04-16T05:37:28.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:28 smithi027 ceph-mon[29206]: Adjusting osd_memory_target on smithi122 to 15937M 2024-04-16T05:37:28.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:28 smithi027 ceph-mon[29206]: osdmap e30: 4 total, 4 up, 4 in 2024-04-16T05:37:28.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:28 smithi027 ceph-mon[29206]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-16T05:37:30.473 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.473+0000 7fcb21600700 1 -- 172.21.15.122:0/130200652 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb1c0fcef0 msgr2=0x7fcb1c0fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:37:30.473 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.473+0000 7fcb21600700 1 --2- 172.21.15.122:0/130200652 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb1c0fcef0 0x7fcb1c0fd320 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c004970 tx=0x7fcb0c009e30 comp rx=0 tx=0).stop 2024-04-16T05:37:30.473 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.473+0000 7fcb21600700 1 -- 172.21.15.122:0/130200652 shutdown_connections 2024-04-16T05:37:30.473 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.473+0000 7fcb21600700 1 --2- 172.21.15.122:0/130200652 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb1c0fe0e0 0x7fcb1c0fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:37:30.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.473+0000 7fcb21600700 1 --2- 172.21.15.122:0/130200652 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb1c0fcef0 0x7fcb1c0fd320 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:37:30.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.473+0000 7fcb21600700 1 -- 172.21.15.122:0/130200652 >> 172.21.15.122:0/130200652 conn(0x7fcb1c0f8530 msgr2=0x7fcb1c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:37:30.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.473+0000 7fcb21600700 1 -- 172.21.15.122:0/130200652 shutdown_connections 2024-04-16T05:37:30.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.473+0000 7fcb21600700 1 -- 172.21.15.122:0/130200652 wait complete. 2024-04-16T05:37:30.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.474+0000 7fcb21600700 1 Processor -- start 2024-04-16T05:37:30.474 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.474+0000 7fcb21600700 1 -- start start 2024-04-16T05:37:30.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.475+0000 7fcb21600700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb1c0fcef0 0x7fcb1c10fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:37:30.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.475+0000 7fcb21600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb1c0fe0e0 0x7fcb1c110170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:37:30.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.475+0000 7fcb21600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c110770 con 0x7fcb1c0fe0e0 2024-04-16T05:37:30.476 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.475+0000 7fcb21600700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1c110fc0 con 0x7fcb1c0fcef0 2024-04-16T05:37:30.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.475+0000 7fcb1aa00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb1c0fcef0 0x7fcb1c10fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:37:30.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.475+0000 7fcb1aa00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb1c0fcef0 0x7fcb1c10fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:55674/0 (socket says 172.21.15.122:55674) 2024-04-16T05:37:30.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.475+0000 7fcb1aa00700 1 -- 172.21.15.122:0/55810715 learned_addr learned my addr 172.21.15.122:0/55810715 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:37:30.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.475+0000 7fcb1aa00700 1 -- 172.21.15.122:0/55810715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb1c0fe0e0 msgr2=0x7fcb1c110170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:37:30.477 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.475+0000 7fcb1aa00700 1 --2- 172.21.15.122:0/55810715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb1c0fe0e0 0x7fcb1c110170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:37:30.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.475+0000 7fcb1aa00700 1 -- 172.21.15.122:0/55810715 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c005040 con 0x7fcb1c0fcef0 2024-04-16T05:37:30.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.475+0000 7fcb1aa00700 1 --2- 172.21.15.122:0/55810715 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb1c0fcef0 0x7fcb1c10fc30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c000c00 tx=0x7fcb0c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:37:30.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.476+0000 7fcb12a00700 1 -- 172.21.15.122:0/55810715 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb0c00e040 con 0x7fcb1c0fcef0 2024-04-16T05:37:30.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.476+0000 7fcb12a00700 1 -- 172.21.15.122:0/55810715 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb0c00bce0 con 0x7fcb1c0fcef0 2024-04-16T05:37:30.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.476+0000 7fcb12a00700 1 -- 172.21.15.122:0/55810715 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb0c01c6a0 con 0x7fcb1c0fcef0 2024-04-16T05:37:30.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.476+0000 7fcb21600700 1 -- 172.21.15.122:0/55810715 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb1c111240 con 0x7fcb1c0fcef0 2024-04-16T05:37:30.478 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.476+0000 7fcb21600700 1 -- 172.21.15.122:0/55810715 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb1c111780 con 0x7fcb1c0fcef0 2024-04-16T05:37:30.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.477+0000 7fcb21600700 1 -- 172.21.15.122:0/55810715 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb1c101a70 con 0x7fcb1c0fcef0 2024-04-16T05:37:30.479 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.478+0000 7fcb12a00700 1 -- 172.21.15.122:0/55810715 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fcb0c013b10 con 0x7fcb1c0fcef0 2024-04-16T05:37:30.480 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.479+0000 7fcb12a00700 1 --2- 172.21.15.122:0/55810715 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fcb000692b0 0x7fcb0006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:37:30.480 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.479+0000 7fcb12a00700 1 -- 172.21.15.122:0/55810715 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3770+0+0 (secure 0 0 0) 0x7fcb0c087e90 con 0x7fcb1c0fcef0 2024-04-16T05:37:30.480 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.480+0000 7fcb1a000700 1 --2- 172.21.15.122:0/55810715 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fcb000692b0 0x7fcb0006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:37:30.480 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.481+0000 7fcb1a000700 1 --2- 172.21.15.122:0/55810715 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fcb000692b0 0x7fcb0006b760 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcb08004ad0 tx=0x7fcb08007d40 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:37:30.484 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.485+0000 7fcb12a00700 1 -- 172.21.15.122:0/55810715 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fcb0c0582d0 con 0x7fcb1c0fcef0 2024-04-16T05:37:30.702 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:30.703+0000 7fcb21600700 1 -- 172.21.15.122:0/55810715 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb1c02b310 con 0x7fcb000692b0 2024-04-16T05:37:31.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:30 smithi122 ceph-mon[35998]: pgmap v128: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-04-16T05:37:31.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:30 smithi027 ceph-mon[29206]: pgmap v128: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-04-16T05:37:31.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:31 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T05:37:31.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:31 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T05:37:31.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:31 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:37:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:31 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T05:37:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:31 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T05:37:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:31 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:37:32.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:32 smithi122 ceph-mon[35998]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:37:32.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:32 smithi122 ceph-mon[35998]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-16T05:37:33.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:32 smithi027 ceph-mon[29206]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:37:33.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:32 smithi027 ceph-mon[29206]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-04-16T05:37:35.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:34 smithi122 ceph-mon[35998]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-16T05:37:35.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:34 smithi122 ceph-mon[35998]: from='client.? 172.21.15.122:0/1359749915' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba8734db-cb7a-4242-b354-8ff1f524f25a"}]: dispatch 2024-04-16T05:37:35.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:34 smithi122 ceph-mon[35998]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba8734db-cb7a-4242-b354-8ff1f524f25a"}]: dispatch 2024-04-16T05:37:35.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:34 smithi122 ceph-mon[35998]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba8734db-cb7a-4242-b354-8ff1f524f25a"}]': finished 2024-04-16T05:37:35.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:34 smithi122 ceph-mon[35998]: osdmap e31: 5 total, 4 up, 5 in 2024-04-16T05:37:35.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:34 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T05:37:35.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:34 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:37:35.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:34 smithi027 ceph-mon[29206]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-16T05:37:35.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:34 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/1359749915' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba8734db-cb7a-4242-b354-8ff1f524f25a"}]: dispatch 2024-04-16T05:37:35.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:34 smithi027 ceph-mon[29206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba8734db-cb7a-4242-b354-8ff1f524f25a"}]: dispatch 2024-04-16T05:37:35.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:34 smithi027 ceph-mon[29206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba8734db-cb7a-4242-b354-8ff1f524f25a"}]': finished 2024-04-16T05:37:35.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:34 smithi027 ceph-mon[29206]: osdmap e31: 5 total, 4 up, 5 in 2024-04-16T05:37:35.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:34 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T05:37:35.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:34 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:37:36.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:35 smithi122 ceph-mon[35998]: from='client.? 172.21.15.122:0/3025615157' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T05:37:36.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:35 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/3025615157' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T05:37:37.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:36 smithi122 ceph-mon[35998]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:37.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:36 smithi027 ceph-mon[29206]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:38.890 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:38 smithi122 ceph-mon[35998]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:39.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:38 smithi027 ceph-mon[29206]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:40.890 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:40 smithi027 ceph-mon[29206]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:41.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:40 smithi122 ceph-mon[35998]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:42.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:42 smithi122 ceph-mon[35998]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:42.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:42 smithi027 ceph-mon[29206]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:44.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:43 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T05:37:44.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:43 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:37:44.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:43 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T05:37:44.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:43 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:37:45.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:44 smithi122 ceph-mon[35998]: Deploying daemon osd.4 on smithi122 2024-04-16T05:37:45.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:44 smithi122 ceph-mon[35998]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:45.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:44 smithi027 ceph-mon[29206]: Deploying daemon osd.4 on smithi122 2024-04-16T05:37:45.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:44 smithi027 ceph-mon[29206]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:46.856 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:46 smithi122 ceph-mon[35998]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:47.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:46 smithi027 ceph-mon[29206]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:48.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:48 smithi122 ceph-mon[35998]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:49.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:48 smithi027 ceph-mon[29206]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:50.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:49 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:37:50.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:49 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:37:50.936 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:50 smithi122 ceph-mon[35998]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:51.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:50 smithi027 ceph-mon[29206]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:52.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:51 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:52.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:51 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:37:52.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:51 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:52.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:51 smithi122 ceph-mon[35998]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:52.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:51 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:52.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:51 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:37:52.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:51 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:52.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:51 smithi027 ceph-mon[29206]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:54.856 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:54 smithi122 ceph-mon[35998]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:55.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:54 smithi027 ceph-mon[29206]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:55.114 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:55.115+0000 7fcb12a00700 1 -- 172.21.15.122:0/55810715 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcb1c02b310 con 0x7fcb000692b0 2024-04-16T05:37:55.114 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 4 on host 'smithi122' 2024-04-16T05:37:55.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:55.117+0000 7fcb21600700 1 -- 172.21.15.122:0/55810715 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fcb000692b0 msgr2=0x7fcb0006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:37:55.116 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:55.117+0000 7fcb21600700 1 --2- 172.21.15.122:0/55810715 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fcb000692b0 0x7fcb0006b760 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcb08004ad0 tx=0x7fcb08007d40 comp rx=0 tx=0).stop 2024-04-16T05:37:55.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:55.117+0000 7fcb21600700 1 -- 172.21.15.122:0/55810715 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb1c0fcef0 msgr2=0x7fcb1c10fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:37:55.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:55.117+0000 7fcb21600700 1 --2- 172.21.15.122:0/55810715 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb1c0fcef0 0x7fcb1c10fc30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c000c00 tx=0x7fcb0c009e30 comp rx=0 tx=0).stop 2024-04-16T05:37:55.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:55.118+0000 7fcb21600700 1 -- 172.21.15.122:0/55810715 shutdown_connections 2024-04-16T05:37:55.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:55.118+0000 7fcb21600700 1 --2- 172.21.15.122:0/55810715 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fcb000692b0 0x7fcb0006b760 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:37:55.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:55.118+0000 7fcb21600700 1 --2- 172.21.15.122:0/55810715 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb1c0fcef0 0x7fcb1c10fc30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:37:55.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:55.118+0000 7fcb21600700 1 --2- 172.21.15.122:0/55810715 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcb1c0fe0e0 0x7fcb1c110170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:37:55.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:55.118+0000 7fcb21600700 1 -- 172.21.15.122:0/55810715 >> 172.21.15.122:0/55810715 conn(0x7fcb1c0f8530 msgr2=0x7fcb1c0f9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:37:55.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:55.118+0000 7fcb21600700 1 -- 172.21.15.122:0/55810715 shutdown_connections 2024-04-16T05:37:55.117 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:37:55.118+0000 7fcb21600700 1 -- 172.21.15.122:0/55810715 wait complete. 2024-04-16T05:37:55.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:55 smithi122 ceph-mon[35998]: from='osd.4 [v2:172.21.15.122:6808/2655596639,v1:172.21.15.122:6809/2655596639]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T05:37:55.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:55 smithi122 ceph-mon[35998]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T05:37:55.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:55 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:55.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:55 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:55.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:55 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:55.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:55 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:55.843 DEBUG:teuthology.orchestra.run.smithi122:osd.4> sudo journalctl -f -n 0 -u ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.4.service 2024-04-16T05:37:55.848 INFO:tasks.cephadm:Deploying osd.5 on smithi122 with /dev/vg_nvme/lv_2... 2024-04-16T05:37:55.849 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-16T05:37:56.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:55 smithi027 ceph-mon[29206]: from='osd.4 [v2:172.21.15.122:6808/2655596639,v1:172.21.15.122:6809/2655596639]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T05:37:56.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:55 smithi027 ceph-mon[29206]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T05:37:56.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:55 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:56.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:55 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:56.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:55 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:56.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:55 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:37:56.343 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi122/config 2024-04-16T05:37:56.745 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:56 smithi122 ceph-mon[35998]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:56.746 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:56 smithi122 ceph-mon[35998]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-16T05:37:56.746 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:56 smithi122 ceph-mon[35998]: osdmap e32: 5 total, 4 up, 5 in 2024-04-16T05:37:56.746 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:56 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T05:37:56.746 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:56 smithi122 ceph-mon[35998]: from='osd.4 [v2:172.21.15.122:6808/2655596639,v1:172.21.15.122:6809/2655596639]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:37:56.746 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:56 smithi122 ceph-mon[35998]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:37:56.746 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:37:56 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[48013]: 2024-04-16T05:37:56.631+0000 7ff014e00700 -1 osd.4 0 waiting for initial osdmap 2024-04-16T05:37:56.746 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:37:56 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[48013]: 2024-04-16T05:37:56.654+0000 7ff00c200700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T05:37:57.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:56 smithi027 ceph-mon[29206]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:57.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:56 smithi027 ceph-mon[29206]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-16T05:37:57.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:56 smithi027 ceph-mon[29206]: osdmap e32: 5 total, 4 up, 5 in 2024-04-16T05:37:57.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:56 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T05:37:57.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:56 smithi027 ceph-mon[29206]: from='osd.4 [v2:172.21.15.122:6808/2655596639,v1:172.21.15.122:6809/2655596639]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:37:57.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:56 smithi027 ceph-mon[29206]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:37:57.869 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:57 smithi122 ceph-mon[35998]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-16T05:37:57.869 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:57 smithi122 ceph-mon[35998]: osdmap e33: 5 total, 4 up, 5 in 2024-04-16T05:37:57.869 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:57 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T05:37:57.869 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:57 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T05:37:58.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:57 smithi027 ceph-mon[29206]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-16T05:37:58.070 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:57 smithi027 ceph-mon[29206]: osdmap e33: 5 total, 4 up, 5 in 2024-04-16T05:37:58.071 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:57 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T05:37:58.071 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:57 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T05:37:58.893 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:58 smithi122 ceph-mon[35998]: purged_snaps scrub starts 2024-04-16T05:37:58.893 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:58 smithi122 ceph-mon[35998]: purged_snaps scrub ok 2024-04-16T05:37:58.893 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:58 smithi122 ceph-mon[35998]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:58.893 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:58 smithi122 ceph-mon[35998]: osd.4 [v2:172.21.15.122:6808/2655596639,v1:172.21.15.122:6809/2655596639] boot 2024-04-16T05:37:58.893 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:58 smithi122 ceph-mon[35998]: osdmap e34: 5 total, 5 up, 5 in 2024-04-16T05:37:58.893 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:58 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T05:37:59.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:58 smithi027 ceph-mon[29206]: purged_snaps scrub starts 2024-04-16T05:37:59.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:58 smithi027 ceph-mon[29206]: purged_snaps scrub ok 2024-04-16T05:37:59.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:58 smithi027 ceph-mon[29206]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-04-16T05:37:59.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:58 smithi027 ceph-mon[29206]: osd.4 [v2:172.21.15.122:6808/2655596639,v1:172.21.15.122:6809/2655596639] boot 2024-04-16T05:37:59.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:58 smithi027 ceph-mon[29206]: osdmap e34: 5 total, 5 up, 5 in 2024-04-16T05:37:59.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:58 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T05:37:59.524 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-16T05:37:59.524 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-16T05:37:59.524 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2024-04-16T05:37:59.524 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2024-04-16T05:37:59.524 INFO:teuthology.orchestra.run.smithi122.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0321132 s, 327 MB/s 2024-04-16T05:37:59.524 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2024-04-16T05:37:59.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:37:59 smithi122 ceph-mon[35998]: osdmap e35: 5 total, 5 up, 5 in 2024-04-16T05:38:00.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:37:59 smithi027 ceph-mon[29206]: osdmap e35: 5 total, 5 up, 5 in 2024-04-16T05:38:00.643 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi122:vg_nvme/lv_2 2024-04-16T05:38:01.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:00 smithi122 ceph-mon[35998]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:01.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:00 smithi027 ceph-mon[29206]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:01.110 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi122/config 2024-04-16T05:38:03.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:02 smithi122 ceph-mon[35998]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:03.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:02 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:03.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:02 smithi027 ceph-mon[29206]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:03.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:02 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:03.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.928+0000 7f0231200700 1 -- 172.21.15.122:0/824943032 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f022c0fffe0 msgr2=0x7f022c1023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:03.928 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.928+0000 7f0231200700 1 --2- 172.21.15.122:0/824943032 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f022c0fffe0 0x7f022c1023c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0218004940 tx=0x7f0218009e30 comp rx=0 tx=0).stop 2024-04-16T05:38:03.928 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:03 smithi122 ceph-mon[35998]: Detected new or changed devices on smithi122 2024-04-16T05:38:03.929 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:03 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:03.929 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:03 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:38:03.929 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:03 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:38:03.929 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:03 smithi122 ceph-mon[35998]: Adjusting osd_memory_target on smithi122 to 7968M 2024-04-16T05:38:03.929 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:03 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:03.929 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:03 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:38:03.929 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:03 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:38:03.929 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:03 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:03.929 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:03 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:38:03.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.928+0000 7f0231200700 1 -- 172.21.15.122:0/824943032 shutdown_connections 2024-04-16T05:38:03.931 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.928+0000 7f0231200700 1 --2- 172.21.15.122:0/824943032 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f022c0fffe0 0x7f022c1023c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:03.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.928+0000 7f0231200700 1 --2- 172.21.15.122:0/824943032 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c066ea0 0x7f022c0ffaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:03.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.928+0000 7f0231200700 1 -- 172.21.15.122:0/824943032 >> 172.21.15.122:0/824943032 conn(0x7f022c0f7450 msgr2=0x7f022c0f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:38:03.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.928+0000 7f0231200700 1 -- 172.21.15.122:0/824943032 shutdown_connections 2024-04-16T05:38:03.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.928+0000 7f0231200700 1 -- 172.21.15.122:0/824943032 wait complete. 2024-04-16T05:38:03.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.929+0000 7f0231200700 1 Processor -- start 2024-04-16T05:38:03.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.929+0000 7f0231200700 1 -- start start 2024-04-16T05:38:03.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.930+0000 7f0231200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f022c066ea0 0x7f022c0fd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:03.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.930+0000 7f0231200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c0fffe0 0x7f022c0fb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:03.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.930+0000 7f0231200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022c0fd8f0 con 0x7f022c0fffe0 2024-04-16T05:38:03.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.930+0000 7f0231200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022c0fbee0 con 0x7f022c066ea0 2024-04-16T05:38:03.932 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.930+0000 7f0229e00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f022c066ea0 0x7f022c0fd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:03.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.930+0000 7f0229e00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f022c066ea0 0x7f022c0fd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49710/0 (socket says 172.21.15.122:49710) 2024-04-16T05:38:03.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.930+0000 7f0229e00700 1 -- 172.21.15.122:0/2523828151 learned_addr learned my addr 172.21.15.122:0/2523828151 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:38:03.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.930+0000 7f0229400700 1 --2- 172.21.15.122:0/2523828151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c0fffe0 0x7f022c0fb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:03.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.931+0000 7f0229e00700 1 -- 172.21.15.122:0/2523828151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c0fffe0 msgr2=0x7f022c0fb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:03.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.931+0000 7f0229e00700 1 --2- 172.21.15.122:0/2523828151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c0fffe0 0x7f022c0fb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:03.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.931+0000 7f0229e00700 1 -- 172.21.15.122:0/2523828151 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0218005040 con 0x7f022c066ea0 2024-04-16T05:38:03.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.931+0000 7f0229400700 1 --2- 172.21.15.122:0/2523828151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c0fffe0 0x7f022c0fb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:38:03.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.931+0000 7f0229e00700 1 --2- 172.21.15.122:0/2523828151 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f022c066ea0 0x7f022c0fd2f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f021c00c8b0 tx=0x7f021c004470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:38:03.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.931+0000 7f0222000700 1 -- 172.21.15.122:0/2523828151 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f021c00a760 con 0x7f022c066ea0 2024-04-16T05:38:03.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.931+0000 7f0222000700 1 -- 172.21.15.122:0/2523828151 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f021c00ad90 con 0x7f022c066ea0 2024-04-16T05:38:03.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.931+0000 7f0231200700 1 -- 172.21.15.122:0/2523828151 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f022c0fc190 con 0x7f022c066ea0 2024-04-16T05:38:03.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.931+0000 7f0222000700 1 -- 172.21.15.122:0/2523828151 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f021c0113a0 con 0x7f022c066ea0 2024-04-16T05:38:03.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.931+0000 7f0231200700 1 -- 172.21.15.122:0/2523828151 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f022c0fc750 con 0x7f022c066ea0 2024-04-16T05:38:03.933 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.933+0000 7f0231200700 1 -- 172.21.15.122:0/2523828151 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f022c04c710 con 0x7f022c066ea0 2024-04-16T05:38:03.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.934+0000 7f0222000700 1 -- 172.21.15.122:0/2523828151 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f021c012400 con 0x7f022c066ea0 2024-04-16T05:38:03.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.935+0000 7f0222000700 1 --2- 172.21.15.122:0/2523828151 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f02100692a0 0x7f021006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:03.934 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.935+0000 7f0222000700 1 -- 172.21.15.122:0/2523828151 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f021c086cb0 con 0x7f022c066ea0 2024-04-16T05:38:03.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.936+0000 7f0229400700 1 --2- 172.21.15.122:0/2523828151 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f02100692a0 0x7f021006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:03.935 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.937+0000 7f0229400700 1 --2- 172.21.15.122:0/2523828151 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f02100692a0 0x7f021006b750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f021800b3b0 tx=0x7f0218009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:38:03.938 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:03.940+0000 7f0222000700 1 -- 172.21.15.122:0/2523828151 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f021c056f40 con 0x7f022c066ea0 2024-04-16T05:38:04.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:03 smithi027 ceph-mon[29206]: Detected new or changed devices on smithi122 2024-04-16T05:38:04.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:03 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:04.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:03 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:38:04.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:03 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:38:04.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:03 smithi027 ceph-mon[29206]: Adjusting osd_memory_target on smithi122 to 7968M 2024-04-16T05:38:04.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:03 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:04.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:03 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:38:04.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:03 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:38:04.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:03 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:04.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:03 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:38:04.159 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:04.160+0000 7f0231200700 1 -- 172.21.15.122:0/2523828151 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f022c101c30 con 0x7f02100692a0 2024-04-16T05:38:05.020 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:04 smithi122 ceph-mon[35998]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:05.020 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:04 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:38:05.020 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:04 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:38:05.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:04 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:05.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:04 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T05:38:05.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:04 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T05:38:05.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:04 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:38:05.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:04 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:38:05.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:04 smithi027 ceph-mon[29206]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:05.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:04 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:38:05.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:04 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:38:05.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:04 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:05.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:04 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-16T05:38:05.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:04 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-16T05:38:05.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:04 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:38:05.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:04 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:38:05.926 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:05 smithi122 ceph-mon[35998]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:38:06.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:05 smithi027 ceph-mon[29206]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:38:07.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:06 smithi122 ceph-mon[35998]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:07.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:06 smithi027 ceph-mon[29206]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:08.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:07 smithi122 ceph-mon[35998]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:08.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:07 smithi122 ceph-mon[35998]: from='client.? 172.21.15.122:0/3285976237' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e2753a7-d194-4103-b586-bfef8cbefa2e"}]: dispatch 2024-04-16T05:38:08.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:07 smithi122 ceph-mon[35998]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e2753a7-d194-4103-b586-bfef8cbefa2e"}]: dispatch 2024-04-16T05:38:08.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:07 smithi027 ceph-mon[29206]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:08.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:07 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/3285976237' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e2753a7-d194-4103-b586-bfef8cbefa2e"}]: dispatch 2024-04-16T05:38:08.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:07 smithi027 ceph-mon[29206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e2753a7-d194-4103-b586-bfef8cbefa2e"}]: dispatch 2024-04-16T05:38:09.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:08 smithi122 ceph-mon[35998]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e2753a7-d194-4103-b586-bfef8cbefa2e"}]': finished 2024-04-16T05:38:09.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:08 smithi122 ceph-mon[35998]: osdmap e36: 6 total, 5 up, 6 in 2024-04-16T05:38:09.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:08 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T05:38:09.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:08 smithi122 ceph-mon[35998]: from='client.? 172.21.15.122:0/4257561941' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T05:38:09.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:08 smithi027 ceph-mon[29206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e2753a7-d194-4103-b586-bfef8cbefa2e"}]': finished 2024-04-16T05:38:09.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:08 smithi027 ceph-mon[29206]: osdmap e36: 6 total, 5 up, 6 in 2024-04-16T05:38:09.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:08 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T05:38:09.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:08 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/4257561941' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-16T05:38:10.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:09 smithi122 ceph-mon[35998]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:10.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:09 smithi027 ceph-mon[29206]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:12.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:12 smithi122 ceph-mon[35998]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:13.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:12 smithi027 ceph-mon[29206]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:15.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:14 smithi122 ceph-mon[35998]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:15.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:14 smithi027 ceph-mon[29206]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:17.008 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:16 smithi122 ceph-mon[35998]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:17.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:16 smithi027 ceph-mon[29206]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:17.980 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:17 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T05:38:17.981 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:17 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:38:18.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:17 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T05:38:18.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:17 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:38:18.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:18 smithi122 ceph-mon[35998]: Deploying daemon osd.5 on smithi122 2024-04-16T05:38:18.849 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:18 smithi122 ceph-mon[35998]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:19.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:18 smithi027 ceph-mon[29206]: Deploying daemon osd.5 on smithi122 2024-04-16T05:38:19.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:18 smithi027 ceph-mon[29206]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:19.913 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:19 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:38:20.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:19 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:38:20.976 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:20 smithi122 ceph-mon[35998]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:21.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:20 smithi027 ceph-mon[29206]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:22.997 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:22 smithi122 ceph-mon[35998]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:23.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:22 smithi027 ceph-mon[29206]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:24.968 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:24 smithi122 ceph-mon[35998]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:24.968 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:24 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:25.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:24 smithi027 ceph-mon[29206]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:25.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:24 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:26.015 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:25 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:26.015 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:25 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:38:26.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:25 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:26.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:25 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:38:26.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:26 smithi122 ceph-mon[35998]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:27.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:26 smithi027 ceph-mon[29206]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:29.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:28 smithi122 ceph-mon[35998]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:29.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:28 smithi122 ceph-mon[35998]: from='osd.5 [v2:172.21.15.122:6816/3490413277,v1:172.21.15.122:6817/3490413277]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T05:38:29.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:28 smithi122 ceph-mon[35998]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T05:38:29.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:28 smithi027 ceph-mon[29206]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:29.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:28 smithi027 ceph-mon[29206]: from='osd.5 [v2:172.21.15.122:6816/3490413277,v1:172.21.15.122:6817/3490413277]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T05:38:29.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:28 smithi027 ceph-mon[29206]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T05:38:29.268 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:29.269+0000 7f0222000700 1 -- 172.21.15.122:0/2523828151 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f022c101c30 con 0x7f02100692a0 2024-04-16T05:38:29.268 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 5 on host 'smithi122' 2024-04-16T05:38:29.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:29.272+0000 7f0231200700 1 -- 172.21.15.122:0/2523828151 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f02100692a0 msgr2=0x7f021006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:29.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:29.272+0000 7f0231200700 1 --2- 172.21.15.122:0/2523828151 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f02100692a0 0x7f021006b750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f021800b3b0 tx=0x7f0218009e30 comp rx=0 tx=0).stop 2024-04-16T05:38:29.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:29.272+0000 7f0231200700 1 -- 172.21.15.122:0/2523828151 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f022c066ea0 msgr2=0x7f022c0fd2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:29.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:29.272+0000 7f0231200700 1 --2- 172.21.15.122:0/2523828151 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f022c066ea0 0x7f022c0fd2f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f021c00c8b0 tx=0x7f021c004470 comp rx=0 tx=0).stop 2024-04-16T05:38:29.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:29.273+0000 7f0231200700 1 -- 172.21.15.122:0/2523828151 shutdown_connections 2024-04-16T05:38:29.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:29.273+0000 7f0231200700 1 --2- 172.21.15.122:0/2523828151 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f02100692a0 0x7f021006b750 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:29.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:29.273+0000 7f0231200700 1 --2- 172.21.15.122:0/2523828151 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f022c066ea0 0x7f022c0fd2f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:29.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:29.273+0000 7f0231200700 1 --2- 172.21.15.122:0/2523828151 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f022c0fffe0 0x7f022c0fb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:29.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:29.273+0000 7f0231200700 1 -- 172.21.15.122:0/2523828151 >> 172.21.15.122:0/2523828151 conn(0x7f022c0f7450 msgr2=0x7f022c0f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:38:29.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:29.273+0000 7f0231200700 1 -- 172.21.15.122:0/2523828151 shutdown_connections 2024-04-16T05:38:29.272 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:29.273+0000 7f0231200700 1 -- 172.21.15.122:0/2523828151 wait complete. 2024-04-16T05:38:29.904 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:29 smithi122 ceph-mon[35998]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-16T05:38:29.905 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:29 smithi122 ceph-mon[35998]: osdmap e37: 6 total, 5 up, 6 in 2024-04-16T05:38:29.905 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T05:38:29.905 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:29 smithi122 ceph-mon[35998]: from='osd.5 [v2:172.21.15.122:6816/3490413277,v1:172.21.15.122:6817/3490413277]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:38:29.905 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:29 smithi122 ceph-mon[35998]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:38:29.905 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:29.905 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:29.905 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:29.906 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:29 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:29.963 DEBUG:teuthology.orchestra.run.smithi122:osd.5> sudo journalctl -f -n 0 -u ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.5.service 2024-04-16T05:38:29.968 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-04-16T05:38:29.968 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph osd stat -f json 2024-04-16T05:38:30.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:29 smithi027 ceph-mon[29206]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-16T05:38:30.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:29 smithi027 ceph-mon[29206]: osdmap e37: 6 total, 5 up, 6 in 2024-04-16T05:38:30.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T05:38:30.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:29 smithi027 ceph-mon[29206]: from='osd.5 [v2:172.21.15.122:6816/3490413277,v1:172.21.15.122:6817/3490413277]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:38:30.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:29 smithi027 ceph-mon[29206]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:38:30.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:30.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:30.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:30.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:29 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:30.492 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:38:31.004 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:30 smithi027 ceph-mon[29206]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:31.004 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:30 smithi027 ceph-mon[29206]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-16T05:38:31.004 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:30 smithi027 ceph-mon[29206]: osdmap e38: 6 total, 5 up, 6 in 2024-04-16T05:38:31.004 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:30 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T05:38:31.005 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:30 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T05:38:31.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:30 smithi122 ceph-mon[35998]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:31.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:30 smithi122 ceph-mon[35998]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2024-04-16T05:38:31.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:30 smithi122 ceph-mon[35998]: osdmap e38: 6 total, 5 up, 6 in 2024-04-16T05:38:31.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:30 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T05:38:31.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:30 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T05:38:31.980 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:31 smithi122 ceph-mon[35998]: purged_snaps scrub starts 2024-04-16T05:38:31.980 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:31 smithi122 ceph-mon[35998]: purged_snaps scrub ok 2024-04-16T05:38:31.980 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:31 smithi122 ceph-mon[35998]: osd.5 [v2:172.21.15.122:6816/3490413277,v1:172.21.15.122:6817/3490413277] boot 2024-04-16T05:38:31.981 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:31 smithi122 ceph-mon[35998]: osdmap e39: 6 total, 6 up, 6 in 2024-04-16T05:38:31.981 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:31 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T05:38:31.990 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:31 smithi027 ceph-mon[29206]: purged_snaps scrub starts 2024-04-16T05:38:31.990 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:31 smithi027 ceph-mon[29206]: purged_snaps scrub ok 2024-04-16T05:38:31.990 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:31 smithi027 ceph-mon[29206]: osd.5 [v2:172.21.15.122:6816/3490413277,v1:172.21.15.122:6817/3490413277] boot 2024-04-16T05:38:31.990 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:31 smithi027 ceph-mon[29206]: osdmap e39: 6 total, 6 up, 6 in 2024-04-16T05:38:31.990 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:31 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T05:38:32.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.801+0000 7f0786400700 1 -- 172.21.15.27:0/1359222880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0780066ac0 msgr2=0x7f07800ffaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:32.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.801+0000 7f0786400700 1 --2- 172.21.15.27:0/1359222880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0780066ac0 0x7f07800ffaa0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0770009ce0 tx=0x7f0770004d10 comp rx=0 tx=0).stop 2024-04-16T05:38:32.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.801+0000 7f0786400700 1 -- 172.21.15.27:0/1359222880 shutdown_connections 2024-04-16T05:38:32.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.801+0000 7f0786400700 1 --2- 172.21.15.27:0/1359222880 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f07800fffe0 0x7f07801023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:32.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.801+0000 7f0786400700 1 --2- 172.21.15.27:0/1359222880 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0780066ac0 0x7f07800ffaa0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:32.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.801+0000 7f0786400700 1 -- 172.21.15.27:0/1359222880 >> 172.21.15.27:0/1359222880 conn(0x7f07800f7450 msgr2=0x7f07800f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:38:32.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.801+0000 7f0786400700 1 -- 172.21.15.27:0/1359222880 shutdown_connections 2024-04-16T05:38:32.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.801+0000 7f0786400700 1 -- 172.21.15.27:0/1359222880 wait complete. 2024-04-16T05:38:32.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.802+0000 7f0786400700 1 Processor -- start 2024-04-16T05:38:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.802+0000 7f0786400700 1 -- start start 2024-04-16T05:38:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.802+0000 7f0786400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0780066ac0 0x7f078010ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.802+0000 7f0786400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07800fffe0 0x7f0780110450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.802+0000 7f0786400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07801111c0 con 0x7f07800fffe0 2024-04-16T05:38:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.802+0000 7f0786400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0780111330 con 0x7f0780066ac0 2024-04-16T05:38:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.803+0000 7f0776a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07800fffe0 0x7f0780110450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.803+0000 7f0776a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07800fffe0 0x7f0780110450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35474/0 (socket says 172.21.15.27:35474) 2024-04-16T05:38:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.803+0000 7f0776a00700 1 -- 172.21.15.27:0/1382182878 learned_addr learned my addr 172.21.15.27:0/1382182878 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:38:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.803+0000 7f077f400700 1 --2- 172.21.15.27:0/1382182878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0780066ac0 0x7f078010ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.803+0000 7f0776a00700 1 -- 172.21.15.27:0/1382182878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0780066ac0 msgr2=0x7f078010ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.803+0000 7f0776a00700 1 --2- 172.21.15.27:0/1382182878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0780066ac0 0x7f078010ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.803+0000 7f0776a00700 1 -- 172.21.15.27:0/1382182878 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0770009940 con 0x7f07800fffe0 2024-04-16T05:38:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.803+0000 7f0776a00700 1 --2- 172.21.15.27:0/1382182878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07800fffe0 0x7f0780110450 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0768002a50 tx=0x7f0768004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:38:32.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.804+0000 7f0777e00700 1 -- 172.21.15.27:0/1382182878 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0768007cc0 con 0x7f07800fffe0 2024-04-16T05:38:32.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.804+0000 7f0777e00700 1 -- 172.21.15.27:0/1382182878 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f076800f040 con 0x7f07800fffe0 2024-04-16T05:38:32.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.804+0000 7f0786400700 1 -- 172.21.15.27:0/1382182878 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07801115b0 con 0x7f07800fffe0 2024-04-16T05:38:32.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.804+0000 7f0777e00700 1 -- 172.21.15.27:0/1382182878 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f076800daf0 con 0x7f07800fffe0 2024-04-16T05:38:32.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.804+0000 7f0786400700 1 -- 172.21.15.27:0/1382182878 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0780111b70 con 0x7f07800fffe0 2024-04-16T05:38:32.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.805+0000 7f0786400700 1 -- 172.21.15.27:0/1382182878 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07800f8fd0 con 0x7f07800fffe0 2024-04-16T05:38:32.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.807+0000 7f0777e00700 1 -- 172.21.15.27:0/1382182878 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f0768009810 con 0x7f07800fffe0 2024-04-16T05:38:32.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.807+0000 7f0777e00700 1 --2- 172.21.15.27:0/1382182878 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f07640692b0 0x7f076406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:32.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.808+0000 7f0777e00700 1 -- 172.21.15.27:0/1382182878 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0768086ca0 con 0x7f07800fffe0 2024-04-16T05:38:32.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.808+0000 7f077f400700 1 --2- 172.21.15.27:0/1382182878 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f07640692b0 0x7f076406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:32.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.809+0000 7f077f400700 1 --2- 172.21.15.27:0/1382182878 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f07640692b0 0x7f076406b760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0770009470 tx=0x7f0770004d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:38:32.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:32.813+0000 7f0777e00700 1 -- 172.21.15.27:0/1382182878 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f07680531f0 con 0x7f07800fffe0 2024-04-16T05:38:33.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:32 smithi122 ceph-mon[35998]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:33.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:32 smithi122 ceph-mon[35998]: osdmap e40: 6 total, 6 up, 6 in 2024-04-16T05:38:33.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:33.030+0000 7f0786400700 1 -- 172.21.15.27:0/1382182878 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f078005f5f0 con 0x7f07800fffe0 2024-04-16T05:38:33.030 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:32 smithi027 ceph-mon[29206]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-04-16T05:38:33.030 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:32 smithi027 ceph-mon[29206]: osdmap e40: 6 total, 6 up, 6 in 2024-04-16T05:38:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:33.031+0000 7f0777e00700 1 -- 172.21.15.27:0/1382182878 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f0768056810 con 0x7f07800fffe0 2024-04-16T05:38:33.031 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:38:33.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:33.034+0000 7f0786400700 1 -- 172.21.15.27:0/1382182878 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f07640692b0 msgr2=0x7f076406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:33.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:33.034+0000 7f0786400700 1 --2- 172.21.15.27:0/1382182878 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f07640692b0 0x7f076406b760 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0770009470 tx=0x7f0770004d10 comp rx=0 tx=0).stop 2024-04-16T05:38:33.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:33.034+0000 7f0786400700 1 -- 172.21.15.27:0/1382182878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07800fffe0 msgr2=0x7f0780110450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:33.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:33.034+0000 7f0786400700 1 --2- 172.21.15.27:0/1382182878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07800fffe0 0x7f0780110450 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0768002a50 tx=0x7f0768004600 comp rx=0 tx=0).stop 2024-04-16T05:38:33.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:33.035+0000 7f0786400700 1 -- 172.21.15.27:0/1382182878 shutdown_connections 2024-04-16T05:38:33.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:33.035+0000 7f0786400700 1 --2- 172.21.15.27:0/1382182878 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f07640692b0 0x7f076406b760 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:33.035+0000 7f0786400700 1 --2- 172.21.15.27:0/1382182878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0780066ac0 0x7f078010ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:33.035+0000 7f0786400700 1 --2- 172.21.15.27:0/1382182878 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07800fffe0 0x7f0780110450 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:33.035+0000 7f0786400700 1 -- 172.21.15.27:0/1382182878 >> 172.21.15.27:0/1382182878 conn(0x7f07800f7450 msgr2=0x7f07800fbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:38:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:33.035+0000 7f0786400700 1 -- 172.21.15.27:0/1382182878 shutdown_connections 2024-04-16T05:38:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:33.035+0000 7f0786400700 1 -- 172.21.15.27:0/1382182878 wait complete. 2024-04-16T05:38:33.834 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1713245910,"num_in_osds":6,"osd_in_since":1713245887,"num_remapped_pgs":0} 2024-04-16T05:38:33.835 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-16T05:38:34.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:33 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/1382182878' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-16T05:38:34.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:33 smithi122 ceph-mon[35998]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:34.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:33 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1382182878' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-16T05:38:34.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:33 smithi027 ceph-mon[29206]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:34.311 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:38:35.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:34 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:38:35.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:34 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:38:36.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:35 smithi122 ceph-mon[35998]: Detected new or changed devices on smithi122 2024-04-16T05:38:36.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:36.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:36.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:38:36.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:38:36.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:38:36.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:35 smithi122 ceph-mon[35998]: Adjusting osd_memory_target on smithi122 to 5312M 2024-04-16T05:38:36.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:36.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:38:36.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:38:36.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:36.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:38:36.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:38:36.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:38:36.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:36.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:35 smithi122 ceph-mon[35998]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:36.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:35 smithi027 ceph-mon[29206]: Detected new or changed devices on smithi122 2024-04-16T05:38:36.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:36.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:36.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:38:36.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:38:36.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:38:36.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:35 smithi027 ceph-mon[29206]: Adjusting osd_memory_target on smithi122 to 5312M 2024-04-16T05:38:36.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:36.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:38:36.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:38:36.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:36.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:38:36.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:38:36.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:38:36.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:38:36.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:35 smithi027 ceph-mon[29206]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:36.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.586+0000 7fa2de000700 1 -- 172.21.15.27:0/3337221490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2d8105090 msgr2=0x7fa2d8105460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:36.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.586+0000 7fa2de000700 1 --2- 172.21.15.27:0/3337221490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2d8105090 0x7fa2d8105460 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8009ce0 tx=0x7fa2c8004d10 comp rx=0 tx=0).stop 2024-04-16T05:38:36.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.586+0000 7fa2de000700 1 -- 172.21.15.27:0/3337221490 shutdown_connections 2024-04-16T05:38:36.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.586+0000 7fa2de000700 1 --2- 172.21.15.27:0/3337221490 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2d8105a30 0x7fa2d80ff0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:36.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.586+0000 7fa2de000700 1 --2- 172.21.15.27:0/3337221490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2d8105090 0x7fa2d8105460 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:36.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.586+0000 7fa2de000700 1 -- 172.21.15.27:0/3337221490 >> 172.21.15.27:0/3337221490 conn(0x7fa2d80fac00 msgr2=0x7fa2d80fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:38:36.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.587+0000 7fa2de000700 1 -- 172.21.15.27:0/3337221490 shutdown_connections 2024-04-16T05:38:36.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.587+0000 7fa2de000700 1 -- 172.21.15.27:0/3337221490 wait complete. 2024-04-16T05:38:36.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.587+0000 7fa2de000700 1 Processor -- start 2024-04-16T05:38:36.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.588+0000 7fa2de000700 1 -- start start 2024-04-16T05:38:36.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.588+0000 7fa2de000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2d8105090 0x7fa2d8115d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:36.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.588+0000 7fa2de000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2d8105a30 0x7fa2d8116290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:36.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.588+0000 7fa2de000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d81168e0 con 0x7fa2d8105090 2024-04-16T05:38:36.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.588+0000 7fa2de000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d810fec0 con 0x7fa2d8105a30 2024-04-16T05:38:36.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.588+0000 7fa2d7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2d8105090 0x7fa2d8115d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.588+0000 7fa2d7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2d8105090 0x7fa2d8115d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60788/0 (socket says 172.21.15.27:60788) 2024-04-16T05:38:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.588+0000 7fa2d7400700 1 -- 172.21.15.27:0/2357837627 learned_addr learned my addr 172.21.15.27:0/2357837627 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:38:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.589+0000 7fa2cea00700 1 --2- 172.21.15.27:0/2357837627 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2d8105a30 0x7fa2d8116290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.589+0000 7fa2d7400700 1 -- 172.21.15.27:0/2357837627 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2d8105a30 msgr2=0x7fa2d8116290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.589+0000 7fa2d7400700 1 --2- 172.21.15.27:0/2357837627 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2d8105a30 0x7fa2d8116290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.589+0000 7fa2d7400700 1 -- 172.21.15.27:0/2357837627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c8009940 con 0x7fa2d8105090 2024-04-16T05:38:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.589+0000 7fa2cea00700 1 --2- 172.21.15.27:0/2357837627 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2d8105a30 0x7fa2d8116290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:38:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.589+0000 7fa2d7400700 1 --2- 172.21.15.27:0/2357837627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2d8105090 0x7fa2d8115d50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8005010 tx=0x7fa2c8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:38:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.589+0000 7fa2cfe00700 1 -- 172.21.15.27:0/2357837627 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2c800e040 con 0x7fa2d8105090 2024-04-16T05:38:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.589+0000 7fa2cfe00700 1 -- 172.21.15.27:0/2357837627 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa2c8008e20 con 0x7fa2d8105090 2024-04-16T05:38:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.589+0000 7fa2cfe00700 1 -- 172.21.15.27:0/2357837627 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2c801c660 con 0x7fa2d8105090 2024-04-16T05:38:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.590+0000 7fa2de000700 1 -- 172.21.15.27:0/2357837627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d8110140 con 0x7fa2d8105090 2024-04-16T05:38:36.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.590+0000 7fa2de000700 1 -- 172.21.15.27:0/2357837627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d8110700 con 0x7fa2d8105090 2024-04-16T05:38:36.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.591+0000 7fa2de000700 1 -- 172.21.15.27:0/2357837627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d81023a0 con 0x7fa2d8105090 2024-04-16T05:38:36.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.592+0000 7fa2cfe00700 1 -- 172.21.15.27:0/2357837627 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fa2c8012a70 con 0x7fa2d8105090 2024-04-16T05:38:36.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.593+0000 7fa2cfe00700 1 --2- 172.21.15.27:0/2357837627 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa2bc0692a0 0x7fa2bc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:36.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.593+0000 7fa2cfe00700 1 -- 172.21.15.27:0/2357837627 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa2c8088ab0 con 0x7fa2d8105090 2024-04-16T05:38:36.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.593+0000 7fa2cea00700 1 --2- 172.21.15.27:0/2357837627 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa2bc0692a0 0x7fa2bc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:36.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.594+0000 7fa2cea00700 1 --2- 172.21.15.27:0/2357837627 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa2bc0692a0 0x7fa2bc06b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0007fd0 tx=0x7fa2c0009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:38:36.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.598+0000 7fa2cfe00700 1 -- 172.21.15.27:0/2357837627 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa2c808ae60 con 0x7fa2d8105090 2024-04-16T05:38:36.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.816+0000 7fa2de000700 1 -- 172.21.15.27:0/2357837627 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2d810c1b0 con 0x7fa2d8105090 2024-04-16T05:38:36.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.818+0000 7fa2cfe00700 1 -- 172.21.15.27:0/2357837627 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11084 (secure 0 0 0) 0x7fa2c8058b10 con 0x7fa2d8105090 2024-04-16T05:38:36.818 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:38:36.818 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":40,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","created":"2024-04-16T05:31:01.142297+0000","modified":"2024-04-16T05:38:31.712493+0000","last_up_change":"2024-04-16T05:38:30.698288+0000","last_in_change":"2024-04-16T05:38:07.718747+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T05:36:52.435427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"4dd7e294-f625-4492-8bd3-18beab34b6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1557847192},{"type":"v1","addr":"172.21.15.27:6803","nonce":1557847192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1557847192},{"type":"v1","addr":"172.21.15.27:6805","nonce":1557847192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1557847192},{"type":"v1","addr":"172.21.15.27:6809","nonce":1557847192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1557847192},{"type":"v1","addr":"172.21.15.27:6807","nonce":1557847192}]},"public_addr":"172.21.15.27:6803/1557847192","cluster_addr":"172.21.15.27:6805/1557847192","heartbeat_back_addr":"172.21.15.27:6809/1557847192","heartbeat_front_addr":"172.21.15.27:6807/1557847192","state":["exists","up"]},{"osd":1,"uuid":"abec61aa-7a28-473b-b9d2-04e8d7de8a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2143934716},{"type":"v1","addr":"172.21.15.27:6811","nonce":2143934716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2143934716},{"type":"v1","addr":"172.21.15.27:6813","nonce":2143934716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2143934716},{"type":"v1","addr":"172.21.15.27:6817","nonce":2143934716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2143934716},{"type":"v1","addr":"172.21.15.27:6815","nonce":2143934716}]},"public_addr":"172.21.15.27:6811/2143934716","cluster_addr":"172.21.15.27:6813/2143934716","heartbeat_back_addr":"172.21.15.27:6817/2143934716","heartbeat_front_addr":"172.21.15.27:6815/2143934716","state":["exists","up"]},{"osd":2,"uuid":"e50778e4-bf1f-45eb-aeab-05e8dad8fa89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3370929967},{"type":"v1","addr":"172.21.15.27:6819","nonce":3370929967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3370929967},{"type":"v1","addr":"172.21.15.27:6821","nonce":3370929967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3370929967},{"type":"v1","addr":"172.21.15.27:6825","nonce":3370929967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3370929967},{"type":"v1","addr":"172.21.15.27:6823","nonce":3370929967}]},"public_addr":"172.21.15.27:6819/3370929967","cluster_addr":"172.21.15.27:6821/3370929967","heartbeat_back_addr":"172.21.15.27:6825/3370929967","heartbeat_front_addr":"172.21.15.27:6823/3370929967","state":["exists","up"]},{"osd":3,"uuid":"8f55dd70-a9d7-4bc1-a79c-028e66571137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":186542313},{"type":"v1","addr":"172.21.15.122:6801","nonce":186542313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":186542313},{"type":"v1","addr":"172.21.15.122:6803","nonce":186542313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":186542313},{"type":"v1","addr":"172.21.15.122:6807","nonce":186542313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":186542313},{"type":"v1","addr":"172.21.15.122:6805","nonce":186542313}]},"public_addr":"172.21.15.122:6801/186542313","cluster_addr":"172.21.15.122:6803/186542313","heartbeat_back_addr":"172.21.15.122:6807/186542313","heartbeat_front_addr":"172.21.15.122:6805/186542313","state":["exists","up"]},{"osd":4,"uuid":"ba8734db-cb7a-4242-b354-8ff1f524f25a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2655596639},{"type":"v1","addr":"172.21.15.122:6809","nonce":2655596639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2655596639},{"type":"v1","addr":"172.21.15.122:6811","nonce":2655596639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2655596639},{"type":"v1","addr":"172.21.15.122:6815","nonce":2655596639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2655596639},{"type":"v1","addr":"172.21.15.122:6813","nonce":2655596639}]},"public_addr":"172.21.15.122:6809/2655596639","cluster_addr":"172.21.15.122:6811/2655596639","heartbeat_back_addr":"172.21.15.122:6815/2655596639","heartbeat_front_addr":"172.21.15.122:6813/2655596639","state":["exists","up"]},{"osd":5,"uuid":"6e2753a7-d194-4103-b586-bfef8cbefa2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":3490413277},{"type":"v1","addr":"172.21.15.122:6817","nonce":3490413277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":3490413277},{"type":"v1","addr":"172.21.15.122:6819","nonce":3490413277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":3490413277},{"type":"v1","addr":"172.21.15.122:6823","nonce":3490413277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":3490413277},{"type":"v1","addr":"172.21.15.122:6821","nonce":3490413277}]},"public_addr":"172.21.15.122:6817/3490413277","cluster_addr":"172.21.15.122:6819/3490413277","heartbeat_back_addr":"172.21.15.122:6823/3490413277","heartbeat_front_addr":"172.21.15.122:6821/3490413277","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:35:28.179537+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:36:08.890364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:36:48.694394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:37:21.969883+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:37:55.840648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:38:29.596986+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/1860000032":"2024-04-17T05:33:49.465983+0000","172.21.15.27:0/602364106":"2024-04-17T05:33:49.465983+0000","172.21.15.27:6801/2700787222":"2024-04-17T05:33:49.465983+0000","172.21.15.27:0/3086153578":"2024-04-17T05:32:17.529651+0000","172.21.15.27:0/4258676557":"2024-04-17T05:32:17.529651+0000","172.21.15.27:0/2242461930":"2024-04-17T05:33:49.465983+0000","172.21.15.27:6801/2779081505":"2024-04-17T05:32:17.529651+0000","172.21.15.27:0/1435069943":"2024-04-17T05:31:34.994394+0000","172.21.15.27:6800/2700787222":"2024-04-17T05:33:49.465983+0000","172.21.15.27:6800/2173136120":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/3170976796":"2024-04-17T05:31:34.994394+0000","172.21.15.27:6801/2173136120":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/3713909498":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/3775595668":"2024-04-17T05:32:17.529651+0000","172.21.15.27:6800/2779081505":"2024-04-17T05:32:17.529651+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T05:38:36.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.821+0000 7fa2de000700 1 -- 172.21.15.27:0/2357837627 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa2bc0692a0 msgr2=0x7fa2bc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:36.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.821+0000 7fa2de000700 1 --2- 172.21.15.27:0/2357837627 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa2bc0692a0 0x7fa2bc06b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0007fd0 tx=0x7fa2c0009040 comp rx=0 tx=0).stop 2024-04-16T05:38:36.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.821+0000 7fa2de000700 1 -- 172.21.15.27:0/2357837627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2d8105090 msgr2=0x7fa2d8115d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:36.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.821+0000 7fa2de000700 1 --2- 172.21.15.27:0/2357837627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2d8105090 0x7fa2d8115d50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8005010 tx=0x7fa2c8004d10 comp rx=0 tx=0).stop 2024-04-16T05:38:36.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.821+0000 7fa2de000700 1 -- 172.21.15.27:0/2357837627 shutdown_connections 2024-04-16T05:38:36.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.821+0000 7fa2de000700 1 --2- 172.21.15.27:0/2357837627 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa2bc0692a0 0x7fa2bc06b750 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:36.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.821+0000 7fa2de000700 1 --2- 172.21.15.27:0/2357837627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2d8105090 0x7fa2d8115d50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:36.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.821+0000 7fa2de000700 1 --2- 172.21.15.27:0/2357837627 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2d8105a30 0x7fa2d8116290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:36.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.821+0000 7fa2de000700 1 -- 172.21.15.27:0/2357837627 >> 172.21.15.27:0/2357837627 conn(0x7fa2d80fac00 msgr2=0x7fa2d80fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:38:36.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.821+0000 7fa2de000700 1 -- 172.21.15.27:0/2357837627 shutdown_connections 2024-04-16T05:38:36.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:36.821+0000 7fa2de000700 1 -- 172.21.15.27:0/2357837627 wait complete. 2024-04-16T05:38:37.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:37 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/2357837627' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T05:38:37.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:37 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/2357837627' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T05:38:37.692 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T05:36:52.435427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-16T05:38:37.693 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-16T05:38:38.195 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:38:38.350 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:38 smithi027 ceph-mon[29206]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:38.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:38 smithi122 ceph-mon[35998]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.435+0000 7f5a62200700 1 -- 172.21.15.27:0/1635347687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a5c072e00 msgr2=0x7f5a5c10d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.435+0000 7f5a62200700 1 --2- 172.21.15.27:0/1635347687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a5c072e00 0x7f5a5c10d5c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5a48004940 tx=0x7f5a48009e30 comp rx=0 tx=0).stop 2024-04-16T05:38:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.435+0000 7f5a62200700 1 -- 172.21.15.27:0/1635347687 shutdown_connections 2024-04-16T05:38:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.435+0000 7f5a62200700 1 --2- 172.21.15.27:0/1635347687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a5c072e00 0x7f5a5c10d5c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.435+0000 7f5a62200700 1 --2- 172.21.15.27:0/1635347687 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a5c0724f0 0x7f5a5c0728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.435+0000 7f5a62200700 1 -- 172.21.15.27:0/1635347687 >> 172.21.15.27:0/1635347687 conn(0x7f5a5c0fab10 msgr2=0x7f5a5c0fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:38:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.435+0000 7f5a62200700 1 -- 172.21.15.27:0/1635347687 shutdown_connections 2024-04-16T05:38:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.435+0000 7f5a62200700 1 -- 172.21.15.27:0/1635347687 wait complete. 2024-04-16T05:38:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.436+0000 7f5a62200700 1 Processor -- start 2024-04-16T05:38:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.436+0000 7f5a62200700 1 -- start start 2024-04-16T05:38:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.437+0000 7f5a62200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a5c0724f0 0x7f5a5c119f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.437+0000 7f5a62200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a5c072e00 0x7f5a5c11a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.437+0000 7f5a62200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c11ab50 con 0x7f5a5c0724f0 2024-04-16T05:38:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.437+0000 7f5a62200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c113e90 con 0x7f5a5c072e00 2024-04-16T05:38:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.437+0000 7f5a60e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a5c0724f0 0x7f5a5c119f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.437+0000 7f5a60e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a5c0724f0 0x7f5a5c119f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60812/0 (socket says 172.21.15.27:60812) 2024-04-16T05:38:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.437+0000 7f5a60e00700 1 -- 172.21.15.27:0/817083459 learned_addr learned my addr 172.21.15.27:0/817083459 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:38:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.437+0000 7f5a5be00700 1 --2- 172.21.15.27:0/817083459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a5c072e00 0x7f5a5c11a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.438+0000 7f5a60e00700 1 -- 172.21.15.27:0/817083459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a5c072e00 msgr2=0x7f5a5c11a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.438+0000 7f5a60e00700 1 --2- 172.21.15.27:0/817083459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a5c072e00 0x7f5a5c11a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.438+0000 7f5a60e00700 1 -- 172.21.15.27:0/817083459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a48005040 con 0x7f5a5c0724f0 2024-04-16T05:38:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.438+0000 7f5a60e00700 1 --2- 172.21.15.27:0/817083459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a5c0724f0 0x7f5a5c119f30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5a5000b790 tx=0x7f5a50009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:38:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.438+0000 7f5a58c00700 1 -- 172.21.15.27:0/817083459 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a5000f770 con 0x7f5a5c0724f0 2024-04-16T05:38:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.438+0000 7f5a58c00700 1 -- 172.21.15.27:0/817083459 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a5000fda0 con 0x7f5a5c0724f0 2024-04-16T05:38:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.438+0000 7f5a62200700 1 -- 172.21.15.27:0/817083459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a5c114110 con 0x7f5a5c0724f0 2024-04-16T05:38:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.438+0000 7f5a58c00700 1 -- 172.21.15.27:0/817083459 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a5000d310 con 0x7f5a5c0724f0 2024-04-16T05:38:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.438+0000 7f5a62200700 1 -- 172.21.15.27:0/817083459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a5c1146d0 con 0x7f5a5c0724f0 2024-04-16T05:38:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.440+0000 7f5a62200700 1 -- 172.21.15.27:0/817083459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a5c073c30 con 0x7f5a5c0724f0 2024-04-16T05:38:40.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.441+0000 7f5a58c00700 1 -- 172.21.15.27:0/817083459 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f5a50010420 con 0x7f5a5c0724f0 2024-04-16T05:38:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.442+0000 7f5a58c00700 1 --2- 172.21.15.27:0/817083459 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f5a440692a0 0x7f5a4406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.442+0000 7f5a58c00700 1 -- 172.21.15.27:0/817083459 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5a50086e30 con 0x7f5a5c0724f0 2024-04-16T05:38:40.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.442+0000 7f5a5be00700 1 --2- 172.21.15.27:0/817083459 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f5a440692a0 0x7f5a4406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:40.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.443+0000 7f5a5be00700 1 --2- 172.21.15.27:0/817083459 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f5a440692a0 0x7f5a4406b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c1157d0 tx=0x7f5a48009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:38:40.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.447+0000 7f5a58c00700 1 -- 172.21.15.27:0/817083459 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f5a50056f10 con 0x7f5a5c0724f0 2024-04-16T05:38:40.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.664+0000 7f5a62200700 1 -- 172.21.15.27:0/817083459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5a5c04bf00 con 0x7f5a5c0724f0 2024-04-16T05:38:40.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.665+0000 7f5a58c00700 1 -- 172.21.15.27:0/817083459 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5a50056aa0 con 0x7f5a5c0724f0 2024-04-16T05:38:40.665 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-16T05:38:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.668+0000 7f5a62200700 1 -- 172.21.15.27:0/817083459 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f5a440692a0 msgr2=0x7f5a4406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.668+0000 7f5a62200700 1 --2- 172.21.15.27:0/817083459 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f5a440692a0 0x7f5a4406b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c1157d0 tx=0x7f5a48009e30 comp rx=0 tx=0).stop 2024-04-16T05:38:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.668+0000 7f5a62200700 1 -- 172.21.15.27:0/817083459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a5c0724f0 msgr2=0x7f5a5c119f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.668+0000 7f5a62200700 1 --2- 172.21.15.27:0/817083459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a5c0724f0 0x7f5a5c119f30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5a5000b790 tx=0x7f5a50009250 comp rx=0 tx=0).stop 2024-04-16T05:38:40.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.669+0000 7f5a62200700 1 -- 172.21.15.27:0/817083459 shutdown_connections 2024-04-16T05:38:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.669+0000 7f5a62200700 1 --2- 172.21.15.27:0/817083459 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f5a440692a0 0x7f5a4406b750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.669+0000 7f5a62200700 1 --2- 172.21.15.27:0/817083459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a5c0724f0 0x7f5a5c119f30 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.669+0000 7f5a62200700 1 --2- 172.21.15.27:0/817083459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a5c072e00 0x7f5a5c11a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.669+0000 7f5a62200700 1 -- 172.21.15.27:0/817083459 >> 172.21.15.27:0/817083459 conn(0x7f5a5c0fab10 msgr2=0x7f5a5c0faee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:38:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.669+0000 7f5a62200700 1 -- 172.21.15.27:0/817083459 shutdown_connections 2024-04-16T05:38:40.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:40.669+0000 7f5a62200700 1 -- 172.21.15.27:0/817083459 wait complete. 2024-04-16T05:38:41.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:40 smithi122 ceph-mon[35998]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:41.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:40 smithi027 ceph-mon[29206]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:41.495 INFO:tasks.cephadm:Setting up client nodes... 2024-04-16T05:38:41.496 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-16T05:38:41.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:41 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/817083459' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T05:38:42.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:41 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/817083459' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T05:38:42.036 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:38:42.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:42 smithi027 ceph-mon[29206]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:43.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:42 smithi122 ceph-mon[35998]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.184+0000 7f0ffce00700 1 -- 172.21.15.27:0/1281256704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ff8070890 msgr2=0x7f0ff8070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.184+0000 7f0ffce00700 1 --2- 172.21.15.27:0/1281256704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ff8070890 0x7f0ff8070d00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0004940 tx=0x7f0fe0009e30 comp rx=0 tx=0).stop 2024-04-16T05:38:44.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.185+0000 7f0ffce00700 1 -- 172.21.15.27:0/1281256704 shutdown_connections 2024-04-16T05:38:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.185+0000 7f0ffce00700 1 --2- 172.21.15.27:0/1281256704 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ff8070890 0x7f0ff8070d00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.185+0000 7f0ffce00700 1 --2- 172.21.15.27:0/1281256704 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ff8071e40 0x7f0ff8070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.185+0000 7f0ffce00700 1 -- 172.21.15.27:0/1281256704 >> 172.21.15.27:0/1281256704 conn(0x7f0ff80f8580 msgr2=0x7f0ff80fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:38:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.185+0000 7f0ffce00700 1 -- 172.21.15.27:0/1281256704 shutdown_connections 2024-04-16T05:38:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.185+0000 7f0ffce00700 1 -- 172.21.15.27:0/1281256704 wait complete. 2024-04-16T05:38:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.186+0000 7f0ffce00700 1 Processor -- start 2024-04-16T05:38:44.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.186+0000 7f0ffce00700 1 -- start start 2024-04-16T05:38:44.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.187+0000 7f0ffce00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ff8070890 0x7f0ff810fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:44.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.187+0000 7f0ffce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ff8071e40 0x7f0ff81100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:44.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.187+0000 7f0ffce00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff8110730 con 0x7f0ff8071e40 2024-04-16T05:38:44.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.187+0000 7f0ffce00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ff81150d0 con 0x7f0ff8070890 2024-04-16T05:38:44.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.187+0000 7f0fefe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ff8071e40 0x7f0ff81100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:44.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.187+0000 7f0fefe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ff8071e40 0x7f0ff81100a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48536/0 (socket says 172.21.15.27:48536) 2024-04-16T05:38:44.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.187+0000 7f0fefe00700 1 -- 172.21.15.27:0/133126393 learned_addr learned my addr 172.21.15.27:0/133126393 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:38:44.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.187+0000 7f0ff5e00700 1 --2- 172.21.15.27:0/133126393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ff8070890 0x7f0ff810fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.187+0000 7f0fefe00700 1 -- 172.21.15.27:0/133126393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ff8070890 msgr2=0x7f0ff810fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.187+0000 7f0fefe00700 1 --2- 172.21.15.27:0/133126393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ff8070890 0x7f0ff810fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.187+0000 7f0fefe00700 1 -- 172.21.15.27:0/133126393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe0005040 con 0x7f0ff8071e40 2024-04-16T05:38:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.187+0000 7f0ff5e00700 1 --2- 172.21.15.27:0/133126393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ff8070890 0x7f0ff810fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:38:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.188+0000 7f0fefe00700 1 --2- 172.21.15.27:0/133126393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ff8071e40 0x7f0ff81100a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0fe000b3b0 tx=0x7f0fe0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:38:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.188+0000 7f0fee000700 1 -- 172.21.15.27:0/133126393 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fe000e040 con 0x7f0ff8071e40 2024-04-16T05:38:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.188+0000 7f0fee000700 1 -- 172.21.15.27:0/133126393 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fe0004290 con 0x7f0ff8071e40 2024-04-16T05:38:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.188+0000 7f0ffce00700 1 -- 172.21.15.27:0/133126393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ff81152d0 con 0x7f0ff8071e40 2024-04-16T05:38:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.188+0000 7f0fee000700 1 -- 172.21.15.27:0/133126393 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fe001c630 con 0x7f0ff8071e40 2024-04-16T05:38:44.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.188+0000 7f0ffce00700 1 -- 172.21.15.27:0/133126393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ff8115840 con 0x7f0ff8071e40 2024-04-16T05:38:44.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.189+0000 7f0ffce00700 1 -- 172.21.15.27:0/133126393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ff804c710 con 0x7f0ff8071e40 2024-04-16T05:38:44.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.190+0000 7f0fee000700 1 -- 172.21.15.27:0/133126393 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f0fe000bbf0 con 0x7f0ff8071e40 2024-04-16T05:38:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.191+0000 7f0fee000700 1 --2- 172.21.15.27:0/133126393 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f0fdc0692b0 0x7f0fdc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.192+0000 7f0fee000700 1 -- 172.21.15.27:0/133126393 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0fe00882d0 con 0x7f0ff8071e40 2024-04-16T05:38:44.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.192+0000 7f0ff5e00700 1 --2- 172.21.15.27:0/133126393 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f0fdc0692b0 0x7f0fdc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:44.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.192+0000 7f0ff5e00700 1 --2- 172.21.15.27:0/133126393 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f0fdc0692b0 0x7f0fdc06b760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8006fd0 tx=0x7f0fe8008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:38:44.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.196+0000 7f0fee000700 1 -- 172.21.15.27:0/133126393 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f0fe00583b0 con 0x7f0ff8071e40 2024-04-16T05:38:44.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.506+0000 7f0ffce00700 1 -- 172.21.15.27:0/133126393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f0ff8115ab0 con 0x7f0ff8071e40 2024-04-16T05:38:44.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.513+0000 7f0fee000700 1 -- 172.21.15.27:0/133126393 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f0fe0057f40 con 0x7f0ff8071e40 2024-04-16T05:38:44.512 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2024-04-16T05:38:44.512 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDkDh5mSUxZHhAAYBexRIIP+eoZTDgaZmBkQw== 2024-04-16T05:38:44.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.515+0000 7f0ffce00700 1 -- 172.21.15.27:0/133126393 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f0fdc0692b0 msgr2=0x7f0fdc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:44.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.515+0000 7f0ffce00700 1 --2- 172.21.15.27:0/133126393 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f0fdc0692b0 0x7f0fdc06b760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8006fd0 tx=0x7f0fe8008040 comp rx=0 tx=0).stop 2024-04-16T05:38:44.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.516+0000 7f0ffce00700 1 -- 172.21.15.27:0/133126393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ff8071e40 msgr2=0x7f0ff81100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:44.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.516+0000 7f0ffce00700 1 --2- 172.21.15.27:0/133126393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ff8071e40 0x7f0ff81100a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0fe000b3b0 tx=0x7f0fe0009e30 comp rx=0 tx=0).stop 2024-04-16T05:38:44.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.516+0000 7f0ffce00700 1 -- 172.21.15.27:0/133126393 shutdown_connections 2024-04-16T05:38:44.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.516+0000 7f0ffce00700 1 --2- 172.21.15.27:0/133126393 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f0fdc0692b0 0x7f0fdc06b760 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:44.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.516+0000 7f0ffce00700 1 --2- 172.21.15.27:0/133126393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ff8070890 0x7f0ff810fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:44.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.516+0000 7f0ffce00700 1 --2- 172.21.15.27:0/133126393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ff8071e40 0x7f0ff81100a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:44.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.516+0000 7f0ffce00700 1 -- 172.21.15.27:0/133126393 >> 172.21.15.27:0/133126393 conn(0x7f0ff80f8580 msgr2=0x7f0ff80f9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:38:44.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.516+0000 7f0ffce00700 1 -- 172.21.15.27:0/133126393 shutdown_connections 2024-04-16T05:38:44.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:44.516+0000 7f0ffce00700 1 -- 172.21.15.27:0/133126393 wait complete. 2024-04-16T05:38:44.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:44 smithi027 ceph-mon[29206]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:44.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:44 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/133126393' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T05:38:44.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:44 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/133126393' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-16T05:38:45.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:44 smithi122 ceph-mon[35998]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:45.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:44 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/133126393' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T05:38:45.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:44 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/133126393' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-16T05:38:45.296 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-16T05:38:45.297 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-16T05:38:45.297 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-16T05:38:45.485 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-16T05:38:45.962 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi122/config 2024-04-16T05:38:46.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:46 smithi122 ceph-mon[35998]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:47.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:46 smithi027 ceph-mon[29206]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:47.993 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.993+0000 7f4afe800700 1 -- 172.21.15.122:0/737035508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4af80ff020 msgr2=0x7f4af80ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:47.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.993+0000 7f4afe800700 1 --2- 172.21.15.122:0/737035508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4af80ff020 0x7f4af80ff430 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8004940 tx=0x7f4ae8009e30 comp rx=0 tx=0).stop 2024-04-16T05:38:47.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.993+0000 7f4afe800700 1 -- 172.21.15.122:0/737035508 shutdown_connections 2024-04-16T05:38:47.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.993+0000 7f4afe800700 1 --2- 172.21.15.122:0/737035508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af81002d0 0x7f4af8100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:47.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.993+0000 7f4afe800700 1 --2- 172.21.15.122:0/737035508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4af80ff020 0x7f4af80ff430 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:47.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.993+0000 7f4afe800700 1 -- 172.21.15.122:0/737035508 >> 172.21.15.122:0/737035508 conn(0x7f4af80fa950 msgr2=0x7f4af80fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:38:47.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.993+0000 7f4afe800700 1 -- 172.21.15.122:0/737035508 shutdown_connections 2024-04-16T05:38:47.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.993+0000 7f4afe800700 1 -- 172.21.15.122:0/737035508 wait complete. 2024-04-16T05:38:47.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.994+0000 7f4afe800700 1 Processor -- start 2024-04-16T05:38:47.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.994+0000 7f4afe800700 1 -- start start 2024-04-16T05:38:47.994 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.994+0000 7f4afe800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af80ff020 0x7f4af810fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:47.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.994+0000 7f4afe800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4af81002d0 0x7f4af8110040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:47.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.994+0000 7f4afe800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af8110640 con 0x7f4af80ff020 2024-04-16T05:38:47.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.994+0000 7f4afe800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af81107b0 con 0x7f4af81002d0 2024-04-16T05:38:47.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.995+0000 7f4af6a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4af81002d0 0x7f4af8110040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:47.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.995+0000 7f4af6a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4af81002d0 0x7f4af8110040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:50344/0 (socket says 172.21.15.122:50344) 2024-04-16T05:38:47.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.995+0000 7f4af6a00700 1 -- 172.21.15.122:0/1532215364 learned_addr learned my addr 172.21.15.122:0/1532215364 (peer_addr_for_me v2:172.21.15.122:0/0) 2024-04-16T05:38:47.995 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.995+0000 7f4af7400700 1 --2- 172.21.15.122:0/1532215364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af80ff020 0x7f4af810fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:47.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.995+0000 7f4af6a00700 1 -- 172.21.15.122:0/1532215364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af80ff020 msgr2=0x7f4af810fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:47.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.995+0000 7f4af6a00700 1 --2- 172.21.15.122:0/1532215364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af80ff020 0x7f4af810fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:47.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.995+0000 7f4af6a00700 1 -- 172.21.15.122:0/1532215364 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae8005040 con 0x7f4af81002d0 2024-04-16T05:38:47.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.996+0000 7f4af6a00700 1 --2- 172.21.15.122:0/1532215364 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4af81002d0 0x7f4af8110040 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4002a50 tx=0x7f4ae40046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:38:47.996 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.996+0000 7f4aef400700 1 -- 172.21.15.122:0/1532215364 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ae4007cc0 con 0x7f4af81002d0 2024-04-16T05:38:47.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.996+0000 7f4aef400700 1 -- 172.21.15.122:0/1532215364 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ae400f040 con 0x7f4af81002d0 2024-04-16T05:38:47.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.996+0000 7f4afe800700 1 -- 172.21.15.122:0/1532215364 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af8115210 con 0x7f4af81002d0 2024-04-16T05:38:47.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.996+0000 7f4aef400700 1 -- 172.21.15.122:0/1532215364 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ae4009640 con 0x7f4af81002d0 2024-04-16T05:38:47.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.996+0000 7f4afe800700 1 -- 172.21.15.122:0/1532215364 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af81157d0 con 0x7f4af81002d0 2024-04-16T05:38:47.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.997+0000 7f4afe800700 1 -- 172.21.15.122:0/1532215364 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af804c710 con 0x7f4af81002d0 2024-04-16T05:38:47.997 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.998+0000 7f4aef400700 1 -- 172.21.15.122:0/1532215364 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f4ae40097a0 con 0x7f4af81002d0 2024-04-16T05:38:47.998 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:47.999+0000 7f4aef400700 1 --2- 172.21.15.122:0/1532215364 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f4adc0692b0 0x7f4adc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:47.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:48.000+0000 7f4aef400700 1 -- 172.21.15.122:0/1532215364 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4ae4086c20 con 0x7f4af81002d0 2024-04-16T05:38:47.999 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:48.000+0000 7f4af7400700 1 --2- 172.21.15.122:0/1532215364 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f4adc0692b0 0x7f4adc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:48.000 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:48.001+0000 7f4af7400700 1 --2- 172.21.15.122:0/1532215364 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f4adc0692b0 0x7f4adc06b760 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4ae800b3b0 tx=0x7f4ae8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:38:48.003 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:48.004+0000 7f4aef400700 1 -- 172.21.15.122:0/1532215364 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f4ae4056c80 con 0x7f4af81002d0 2024-04-16T05:38:48.294 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:48.295+0000 7f4afe800700 1 -- 172.21.15.122:0/1532215364 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f4af8115c50 con 0x7f4af81002d0 2024-04-16T05:38:48.305 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:48.306+0000 7f4aef400700 1 -- 172.21.15.122:0/1532215364 <== mon.1 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f4ae4056810 con 0x7f4af81002d0 2024-04-16T05:38:48.305 INFO:teuthology.orchestra.run.smithi122.stdout:[client.1] 2024-04-16T05:38:48.305 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQDoDh5mcdDNERAAE7sp506lynQKkb/M8vsS/g== 2024-04-16T05:38:48.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:48.309+0000 7f4afe800700 1 -- 172.21.15.122:0/1532215364 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f4adc0692b0 msgr2=0x7f4adc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:48.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:48.309+0000 7f4afe800700 1 --2- 172.21.15.122:0/1532215364 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f4adc0692b0 0x7f4adc06b760 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4ae800b3b0 tx=0x7f4ae8009e30 comp rx=0 tx=0).stop 2024-04-16T05:38:48.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:48.309+0000 7f4afe800700 1 -- 172.21.15.122:0/1532215364 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4af81002d0 msgr2=0x7f4af8110040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:48.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:48.309+0000 7f4afe800700 1 --2- 172.21.15.122:0/1532215364 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4af81002d0 0x7f4af8110040 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4002a50 tx=0x7f4ae40046b0 comp rx=0 tx=0).stop 2024-04-16T05:38:48.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:48.309+0000 7f4afe800700 1 -- 172.21.15.122:0/1532215364 shutdown_connections 2024-04-16T05:38:48.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:48.309+0000 7f4afe800700 1 --2- 172.21.15.122:0/1532215364 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f4adc0692b0 0x7f4adc06b760 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:48.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:48.309+0000 7f4afe800700 1 --2- 172.21.15.122:0/1532215364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4af80ff020 0x7f4af810fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:48.308 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:48.309+0000 7f4afe800700 1 --2- 172.21.15.122:0/1532215364 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4af81002d0 0x7f4af8110040 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:48.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:48.309+0000 7f4afe800700 1 -- 172.21.15.122:0/1532215364 >> 172.21.15.122:0/1532215364 conn(0x7f4af80fa950 msgr2=0x7f4af80fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:38:48.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:48.309+0000 7f4afe800700 1 -- 172.21.15.122:0/1532215364 shutdown_connections 2024-04-16T05:38:48.309 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:38:48.309+0000 7f4afe800700 1 -- 172.21.15.122:0/1532215364 wait complete. 2024-04-16T05:38:48.926 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:48 smithi122 ceph-mon[35998]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:48.927 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:48 smithi122 ceph-mon[35998]: from='client.? 172.21.15.122:0/1532215364' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T05:38:48.927 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:48 smithi122 ceph-mon[35998]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T05:38:48.927 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:48 smithi122 ceph-mon[35998]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-16T05:38:49.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:48 smithi027 ceph-mon[29206]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:49.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:48 smithi027 ceph-mon[29206]: from='client.? 172.21.15.122:0/1532215364' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T05:38:49.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:48 smithi027 ceph-mon[29206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-16T05:38:49.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:48 smithi027 ceph-mon[29206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-16T05:38:49.127 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-04-16T05:38:49.128 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-16T05:38:49.128 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-16T05:38:49.267 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-16T05:38:49.267 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-16T05:38:49.267 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph mgr dump --format=json 2024-04-16T05:38:49.790 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:38:49.940 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:49 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:38:50.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:49 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:38:50.986 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:50 smithi027 ceph-mon[29206]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:51.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:50 smithi122 ceph-mon[35998]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:51.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.906+0000 7fdfcf600700 1 -- 172.21.15.27:0/1552805805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfc8102e20 msgr2=0x7fdfc81031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:51.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.906+0000 7fdfcf600700 1 --2- 172.21.15.27:0/1552805805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfc8102e20 0x7fdfc81031f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0004940 tx=0x7fdfb0009e30 comp rx=0 tx=0).stop 2024-04-16T05:38:51.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.906+0000 7fdfcf600700 1 -- 172.21.15.27:0/1552805805 shutdown_connections 2024-04-16T05:38:51.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.906+0000 7fdfcf600700 1 --2- 172.21.15.27:0/1552805805 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfc81037c0 0x7fdfc80fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:51.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.906+0000 7fdfcf600700 1 --2- 172.21.15.27:0/1552805805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfc8102e20 0x7fdfc81031f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:51.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.906+0000 7fdfcf600700 1 -- 172.21.15.27:0/1552805805 >> 172.21.15.27:0/1552805805 conn(0x7fdfc80f89b0 msgr2=0x7fdfc80fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:38:51.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.906+0000 7fdfcf600700 1 -- 172.21.15.27:0/1552805805 shutdown_connections 2024-04-16T05:38:51.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.906+0000 7fdfcf600700 1 -- 172.21.15.27:0/1552805805 wait complete. 2024-04-16T05:38:51.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.907+0000 7fdfcf600700 1 Processor -- start 2024-04-16T05:38:51.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.908+0000 7fdfcf600700 1 -- start start 2024-04-16T05:38:51.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.908+0000 7fdfcf600700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfc8102e20 0x7fdfc810a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:51.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.908+0000 7fdfcf600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfc81037c0 0x7fdfc810ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:51.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.908+0000 7fdfcf600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc810e880 con 0x7fdfc81037c0 2024-04-16T05:38:51.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.908+0000 7fdfcf600700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc810b170 con 0x7fdfc8102e20 2024-04-16T05:38:51.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.909+0000 7fdfbfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfc81037c0 0x7fdfc810ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:51.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.909+0000 7fdfbfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfc81037c0 0x7fdfc810ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48552/0 (socket says 172.21.15.27:48552) 2024-04-16T05:38:51.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.909+0000 7fdfbfe00700 1 -- 172.21.15.27:0/3802820146 learned_addr learned my addr 172.21.15.27:0/3802820146 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:38:51.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.909+0000 7fdfccc00700 1 --2- 172.21.15.27:0/3802820146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfc8102e20 0x7fdfc810a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:51.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.909+0000 7fdfbfe00700 1 -- 172.21.15.27:0/3802820146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfc8102e20 msgr2=0x7fdfc810a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:51.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.909+0000 7fdfbfe00700 1 --2- 172.21.15.27:0/3802820146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfc8102e20 0x7fdfc810a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:51.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.909+0000 7fdfbfe00700 1 -- 172.21.15.27:0/3802820146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb0005040 con 0x7fdfc81037c0 2024-04-16T05:38:51.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.910+0000 7fdfbfe00700 1 --2- 172.21.15.27:0/3802820146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfc81037c0 0x7fdfc810ac30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdfb8002a50 tx=0x7fdfb8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:38:51.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.910+0000 7fdfc5600700 1 -- 172.21.15.27:0/3802820146 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfb8007cc0 con 0x7fdfc81037c0 2024-04-16T05:38:51.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.910+0000 7fdfcf600700 1 -- 172.21.15.27:0/3802820146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc810b450 con 0x7fdfc81037c0 2024-04-16T05:38:51.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.910+0000 7fdfc5600700 1 -- 172.21.15.27:0/3802820146 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdfb800f040 con 0x7fdfc81037c0 2024-04-16T05:38:51.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.910+0000 7fdfc5600700 1 -- 172.21.15.27:0/3802820146 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfb800d9c0 con 0x7fdfc81037c0 2024-04-16T05:38:51.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.910+0000 7fdfcf600700 1 -- 172.21.15.27:0/3802820146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc811e7b0 con 0x7fdfc81037c0 2024-04-16T05:38:51.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.911+0000 7fdfcf600700 1 -- 172.21.15.27:0/3802820146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfc8106ac0 con 0x7fdfc81037c0 2024-04-16T05:38:51.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.912+0000 7fdfc5600700 1 -- 172.21.15.27:0/3802820146 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fdfb8009730 con 0x7fdfc81037c0 2024-04-16T05:38:51.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.913+0000 7fdfc5600700 1 --2- 172.21.15.27:0/3802820146 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fdfa80692b0 0x7fdfa806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:51.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.913+0000 7fdfc5600700 1 -- 172.21.15.27:0/3802820146 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdfb8086c90 con 0x7fdfc81037c0 2024-04-16T05:38:51.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.914+0000 7fdfccc00700 1 --2- 172.21.15.27:0/3802820146 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fdfa80692b0 0x7fdfa806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:51.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.915+0000 7fdfccc00700 1 --2- 172.21.15.27:0/3802820146 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fdfa80692b0 0x7fdfa806b760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdfb000b3b0 tx=0x7fdfb0009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:38:51.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:51.918+0000 7fdfc5600700 1 -- 172.21.15.27:0/3802820146 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fdfb80531e0 con 0x7fdfc81037c0 2024-04-16T05:38:52.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:52.199+0000 7fdfcf600700 1 -- 172.21.15.27:0/3802820146 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fdfc80fe5c0 con 0x7fdfc81037c0 2024-04-16T05:38:52.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:52.203+0000 7fdfc5600700 1 -- 172.21.15.27:0/3802820146 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167177 (secure 0 0 0) 0x7fdfb8056800 con 0x7fdfc81037c0 2024-04-16T05:38:52.206 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:38:52.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:52.209+0000 7fdfcf600700 1 -- 172.21.15.27:0/3802820146 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fdfa80692b0 msgr2=0x7fdfa806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:52.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:52.209+0000 7fdfcf600700 1 --2- 172.21.15.27:0/3802820146 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fdfa80692b0 0x7fdfa806b760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdfb000b3b0 tx=0x7fdfb0009e30 comp rx=0 tx=0).stop 2024-04-16T05:38:52.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:52.209+0000 7fdfcf600700 1 -- 172.21.15.27:0/3802820146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfc81037c0 msgr2=0x7fdfc810ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:52.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:52.209+0000 7fdfcf600700 1 --2- 172.21.15.27:0/3802820146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfc81037c0 0x7fdfc810ac30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdfb8002a50 tx=0x7fdfb8004600 comp rx=0 tx=0).stop 2024-04-16T05:38:52.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:52.209+0000 7fdfcf600700 1 -- 172.21.15.27:0/3802820146 shutdown_connections 2024-04-16T05:38:52.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:52.209+0000 7fdfcf600700 1 --2- 172.21.15.27:0/3802820146 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fdfa80692b0 0x7fdfa806b760 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:52.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:52.209+0000 7fdfcf600700 1 --2- 172.21.15.27:0/3802820146 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfc8102e20 0x7fdfc810a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:52.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:52.209+0000 7fdfcf600700 1 --2- 172.21.15.27:0/3802820146 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfc81037c0 0x7fdfc810ac30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:52.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:52.209+0000 7fdfcf600700 1 -- 172.21.15.27:0/3802820146 >> 172.21.15.27:0/3802820146 conn(0x7fdfc80f89b0 msgr2=0x7fdfc8107410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:38:52.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:52.210+0000 7fdfcf600700 1 -- 172.21.15.27:0/3802820146 shutdown_connections 2024-04-16T05:38:52.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:52.210+0000 7fdfcf600700 1 -- 172.21.15.27:0/3802820146 wait complete. 2024-04-16T05:38:52.983 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:52 smithi027 ceph-mon[29206]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:52.983 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:52 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/3802820146' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-16T05:38:53.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:52 smithi122 ceph-mon[35998]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:53.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:52 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/3802820146' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-16T05:38:53.062 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"flags":0,"active_gid":14212,"active_name":"smithi027.qgzypi","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":2203377971},{"type":"v1","addr":"172.21.15.27:6801","nonce":2203377971}]},"active_addr":"172.21.15.27:6801/2203377971","active_change":"2024-04-16T05:33:49.466431+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14234,"name":"smithi122.sgokkq","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.27:8443/","prometheus":"http://172.21.15.27:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":2938698524}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":99903288}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":3077390652}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":4061328916}]}]} 2024-04-16T05:38:53.067 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-16T05:38:53.067 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-16T05:38:53.067 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-16T05:38:53.598 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:38:54.951 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:54 smithi027 ceph-mon[29206]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:55.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:54 smithi122 ceph-mon[35998]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:55.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.783+0000 7f08f9c00700 1 -- 172.21.15.27:0/483517457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08f40fe830 msgr2=0x7f08f4106e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:55.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.783+0000 7f08f9c00700 1 --2- 172.21.15.27:0/483517457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08f40fe830 0x7f08f4106e70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f08e0004940 tx=0x7f08e0009e30 comp rx=0 tx=0).stop 2024-04-16T05:38:55.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.783+0000 7f08f9c00700 1 -- 172.21.15.27:0/483517457 shutdown_connections 2024-04-16T05:38:55.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.783+0000 7f08f9c00700 1 --2- 172.21.15.27:0/483517457 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08f40fe830 0x7f08f4106e70 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:55.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.783+0000 7f08f9c00700 1 --2- 172.21.15.27:0/483517457 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08f40fde90 0x7f08f40fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:55.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.783+0000 7f08f9c00700 1 -- 172.21.15.27:0/483517457 >> 172.21.15.27:0/483517457 conn(0x7f08f40f7900 msgr2=0x7f08f40f9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:38:55.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.783+0000 7f08f9c00700 1 -- 172.21.15.27:0/483517457 shutdown_connections 2024-04-16T05:38:55.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.783+0000 7f08f9c00700 1 -- 172.21.15.27:0/483517457 wait complete. 2024-04-16T05:38:55.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.784+0000 7f08f9c00700 1 Processor -- start 2024-04-16T05:38:55.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.784+0000 7f08f9c00700 1 -- start start 2024-04-16T05:38:55.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.784+0000 7f08f9c00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08f40fde90 0x7f08f4115b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:55.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.785+0000 7f08f9c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08f40fe830 0x7f08f41160a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:55.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.785+0000 7f08f9c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f41166f0 con 0x7f08f40fe830 2024-04-16T05:38:55.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.785+0000 7f08f9c00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f410fcd0 con 0x7f08f40fde90 2024-04-16T05:38:55.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.785+0000 7f08f2000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08f40fe830 0x7f08f41160a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:55.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.785+0000 7f08f2000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08f40fe830 0x7f08f41160a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50612/0 (socket says 172.21.15.27:50612) 2024-04-16T05:38:55.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.785+0000 7f08f2000700 1 -- 172.21.15.27:0/2325639375 learned_addr learned my addr 172.21.15.27:0/2325639375 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:38:55.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.785+0000 7f08f2a00700 1 --2- 172.21.15.27:0/2325639375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08f40fde90 0x7f08f4115b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:55.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.785+0000 7f08f2000700 1 -- 172.21.15.27:0/2325639375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08f40fde90 msgr2=0x7f08f4115b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:55.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.785+0000 7f08f2000700 1 --2- 172.21.15.27:0/2325639375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08f40fde90 0x7f08f4115b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:55.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.785+0000 7f08f2000700 1 -- 172.21.15.27:0/2325639375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e0005040 con 0x7f08f40fe830 2024-04-16T05:38:55.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.786+0000 7f08f2000700 1 --2- 172.21.15.27:0/2325639375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08f40fe830 0x7f08f41160a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f08e0009180 tx=0x7f08e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:38:55.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.786+0000 7f08eaa00700 1 -- 172.21.15.27:0/2325639375 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08e000e040 con 0x7f08f40fe830 2024-04-16T05:38:55.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.786+0000 7f08eaa00700 1 -- 172.21.15.27:0/2325639375 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f08e0013420 con 0x7f08f40fe830 2024-04-16T05:38:55.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.786+0000 7f08f9c00700 1 -- 172.21.15.27:0/2325639375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08f410ff50 con 0x7f08f40fe830 2024-04-16T05:38:55.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.786+0000 7f08eaa00700 1 -- 172.21.15.27:0/2325639375 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08e001c630 con 0x7f08f40fe830 2024-04-16T05:38:55.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.786+0000 7f08f9c00700 1 -- 172.21.15.27:0/2325639375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f4110490 con 0x7f08f40fe830 2024-04-16T05:38:55.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.787+0000 7f08f9c00700 1 -- 172.21.15.27:0/2325639375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08f41068e0 con 0x7f08f40fe830 2024-04-16T05:38:55.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.788+0000 7f08eaa00700 1 -- 172.21.15.27:0/2325639375 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f08e0013590 con 0x7f08f40fe830 2024-04-16T05:38:55.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.789+0000 7f08eaa00700 1 --2- 172.21.15.27:0/2325639375 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f08d80692a0 0x7f08d806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:55.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.789+0000 7f08eaa00700 1 -- 172.21.15.27:0/2325639375 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f08e00882c0 con 0x7f08f40fe830 2024-04-16T05:38:55.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.790+0000 7f08f2a00700 1 --2- 172.21.15.27:0/2325639375 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f08d80692a0 0x7f08d806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:55.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.790+0000 7f08f2a00700 1 --2- 172.21.15.27:0/2325639375 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f08d80692a0 0x7f08d806b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f08e4007fd0 tx=0x7f08e4009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:38:55.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:55.794+0000 7f08eaa00700 1 -- 172.21.15.27:0/2325639375 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f08e0058320 con 0x7f08f40fe830 2024-04-16T05:38:56.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:56.011+0000 7f08f9c00700 1 -- 172.21.15.27:0/2325639375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08f41111b0 con 0x7f08f40fe830 2024-04-16T05:38:56.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:56.014+0000 7f08eaa00700 1 -- 172.21.15.27:0/2325639375 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11084 (secure 0 0 0) 0x7f08e0057eb0 con 0x7f08f40fe830 2024-04-16T05:38:56.013 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:38:56.013 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":40,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","created":"2024-04-16T05:31:01.142297+0000","modified":"2024-04-16T05:38:31.712493+0000","last_up_change":"2024-04-16T05:38:30.698288+0000","last_in_change":"2024-04-16T05:38:07.718747+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T05:36:52.435427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"4dd7e294-f625-4492-8bd3-18beab34b6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1557847192},{"type":"v1","addr":"172.21.15.27:6803","nonce":1557847192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1557847192},{"type":"v1","addr":"172.21.15.27:6805","nonce":1557847192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1557847192},{"type":"v1","addr":"172.21.15.27:6809","nonce":1557847192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1557847192},{"type":"v1","addr":"172.21.15.27:6807","nonce":1557847192}]},"public_addr":"172.21.15.27:6803/1557847192","cluster_addr":"172.21.15.27:6805/1557847192","heartbeat_back_addr":"172.21.15.27:6809/1557847192","heartbeat_front_addr":"172.21.15.27:6807/1557847192","state":["exists","up"]},{"osd":1,"uuid":"abec61aa-7a28-473b-b9d2-04e8d7de8a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2143934716},{"type":"v1","addr":"172.21.15.27:6811","nonce":2143934716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2143934716},{"type":"v1","addr":"172.21.15.27:6813","nonce":2143934716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2143934716},{"type":"v1","addr":"172.21.15.27:6817","nonce":2143934716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2143934716},{"type":"v1","addr":"172.21.15.27:6815","nonce":2143934716}]},"public_addr":"172.21.15.27:6811/2143934716","cluster_addr":"172.21.15.27:6813/2143934716","heartbeat_back_addr":"172.21.15.27:6817/2143934716","heartbeat_front_addr":"172.21.15.27:6815/2143934716","state":["exists","up"]},{"osd":2,"uuid":"e50778e4-bf1f-45eb-aeab-05e8dad8fa89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3370929967},{"type":"v1","addr":"172.21.15.27:6819","nonce":3370929967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3370929967},{"type":"v1","addr":"172.21.15.27:6821","nonce":3370929967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3370929967},{"type":"v1","addr":"172.21.15.27:6825","nonce":3370929967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3370929967},{"type":"v1","addr":"172.21.15.27:6823","nonce":3370929967}]},"public_addr":"172.21.15.27:6819/3370929967","cluster_addr":"172.21.15.27:6821/3370929967","heartbeat_back_addr":"172.21.15.27:6825/3370929967","heartbeat_front_addr":"172.21.15.27:6823/3370929967","state":["exists","up"]},{"osd":3,"uuid":"8f55dd70-a9d7-4bc1-a79c-028e66571137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":186542313},{"type":"v1","addr":"172.21.15.122:6801","nonce":186542313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":186542313},{"type":"v1","addr":"172.21.15.122:6803","nonce":186542313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":186542313},{"type":"v1","addr":"172.21.15.122:6807","nonce":186542313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":186542313},{"type":"v1","addr":"172.21.15.122:6805","nonce":186542313}]},"public_addr":"172.21.15.122:6801/186542313","cluster_addr":"172.21.15.122:6803/186542313","heartbeat_back_addr":"172.21.15.122:6807/186542313","heartbeat_front_addr":"172.21.15.122:6805/186542313","state":["exists","up"]},{"osd":4,"uuid":"ba8734db-cb7a-4242-b354-8ff1f524f25a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2655596639},{"type":"v1","addr":"172.21.15.122:6809","nonce":2655596639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2655596639},{"type":"v1","addr":"172.21.15.122:6811","nonce":2655596639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2655596639},{"type":"v1","addr":"172.21.15.122:6815","nonce":2655596639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2655596639},{"type":"v1","addr":"172.21.15.122:6813","nonce":2655596639}]},"public_addr":"172.21.15.122:6809/2655596639","cluster_addr":"172.21.15.122:6811/2655596639","heartbeat_back_addr":"172.21.15.122:6815/2655596639","heartbeat_front_addr":"172.21.15.122:6813/2655596639","state":["exists","up"]},{"osd":5,"uuid":"6e2753a7-d194-4103-b586-bfef8cbefa2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":3490413277},{"type":"v1","addr":"172.21.15.122:6817","nonce":3490413277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":3490413277},{"type":"v1","addr":"172.21.15.122:6819","nonce":3490413277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":3490413277},{"type":"v1","addr":"172.21.15.122:6823","nonce":3490413277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":3490413277},{"type":"v1","addr":"172.21.15.122:6821","nonce":3490413277}]},"public_addr":"172.21.15.122:6817/3490413277","cluster_addr":"172.21.15.122:6819/3490413277","heartbeat_back_addr":"172.21.15.122:6823/3490413277","heartbeat_front_addr":"172.21.15.122:6821/3490413277","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:35:28.179537+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:36:08.890364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:36:48.694394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:37:21.969883+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:37:55.840648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:38:29.596986+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/1860000032":"2024-04-17T05:33:49.465983+0000","172.21.15.27:0/602364106":"2024-04-17T05:33:49.465983+0000","172.21.15.27:6801/2700787222":"2024-04-17T05:33:49.465983+0000","172.21.15.27:0/3086153578":"2024-04-17T05:32:17.529651+0000","172.21.15.27:0/4258676557":"2024-04-17T05:32:17.529651+0000","172.21.15.27:0/2242461930":"2024-04-17T05:33:49.465983+0000","172.21.15.27:6801/2779081505":"2024-04-17T05:32:17.529651+0000","172.21.15.27:0/1435069943":"2024-04-17T05:31:34.994394+0000","172.21.15.27:6800/2700787222":"2024-04-17T05:33:49.465983+0000","172.21.15.27:6800/2173136120":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/3170976796":"2024-04-17T05:31:34.994394+0000","172.21.15.27:6801/2173136120":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/3713909498":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/3775595668":"2024-04-17T05:32:17.529651+0000","172.21.15.27:6800/2779081505":"2024-04-17T05:32:17.529651+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T05:38:56.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:56.017+0000 7f08f9c00700 1 -- 172.21.15.27:0/2325639375 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f08d80692a0 msgr2=0x7f08d806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:56.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:56.017+0000 7f08f9c00700 1 --2- 172.21.15.27:0/2325639375 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f08d80692a0 0x7f08d806b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f08e4007fd0 tx=0x7f08e4009040 comp rx=0 tx=0).stop 2024-04-16T05:38:56.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:56.017+0000 7f08f9c00700 1 -- 172.21.15.27:0/2325639375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08f40fe830 msgr2=0x7f08f41160a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:56.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:56.017+0000 7f08f9c00700 1 --2- 172.21.15.27:0/2325639375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08f40fe830 0x7f08f41160a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f08e0009180 tx=0x7f08e0009e30 comp rx=0 tx=0).stop 2024-04-16T05:38:56.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:56.017+0000 7f08f9c00700 1 -- 172.21.15.27:0/2325639375 shutdown_connections 2024-04-16T05:38:56.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:56.017+0000 7f08f9c00700 1 --2- 172.21.15.27:0/2325639375 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f08d80692a0 0x7f08d806b750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:56.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:56.017+0000 7f08f9c00700 1 --2- 172.21.15.27:0/2325639375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08f40fde90 0x7f08f4115b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:56.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:56.017+0000 7f08f9c00700 1 --2- 172.21.15.27:0/2325639375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08f40fe830 0x7f08f41160a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:56.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:56.017+0000 7f08f9c00700 1 -- 172.21.15.27:0/2325639375 >> 172.21.15.27:0/2325639375 conn(0x7f08f40f7900 msgr2=0x7f08f40fd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:38:56.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:56.017+0000 7f08f9c00700 1 -- 172.21.15.27:0/2325639375 shutdown_connections 2024-04-16T05:38:56.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:56.018+0000 7f08f9c00700 1 -- 172.21.15.27:0/2325639375 wait complete. 2024-04-16T05:38:56.875 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-16T05:38:56.876 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-16T05:38:56.973 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:56 smithi027 ceph-mon[29206]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:56.973 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:56 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/2325639375' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T05:38:57.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:56 smithi122 ceph-mon[35998]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:57.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:56 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/2325639375' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T05:38:57.391 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:38:59.020 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:38:58 smithi027 ceph-mon[29206]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:59.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:38:58 smithi122 ceph-mon[35998]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:38:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.564+0000 7f6faf800700 1 -- 172.21.15.27:0/254414929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fa81059b0 msgr2=0x7f6fa80ff020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.564+0000 7f6faf800700 1 --2- 172.21.15.27:0/254414929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fa81059b0 0x7f6fa80ff020 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6f94004940 tx=0x7f6f94009e30 comp rx=0 tx=0).stop 2024-04-16T05:38:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.565+0000 7f6faf800700 1 -- 172.21.15.27:0/254414929 shutdown_connections 2024-04-16T05:38:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.565+0000 7f6faf800700 1 --2- 172.21.15.27:0/254414929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fa81059b0 0x7f6fa80ff020 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.565+0000 7f6faf800700 1 --2- 172.21.15.27:0/254414929 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fa8105010 0x7f6fa81053e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.565+0000 7f6faf800700 1 -- 172.21.15.27:0/254414929 >> 172.21.15.27:0/254414929 conn(0x7f6fa80faba0 msgr2=0x7f6fa80fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:38:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.565+0000 7f6faf800700 1 -- 172.21.15.27:0/254414929 shutdown_connections 2024-04-16T05:38:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.565+0000 7f6faf800700 1 -- 172.21.15.27:0/254414929 wait complete. 2024-04-16T05:38:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.566+0000 7f6faf800700 1 Processor -- start 2024-04-16T05:38:59.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.566+0000 7f6faf800700 1 -- start start 2024-04-16T05:38:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.566+0000 7f6faf800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fa8105010 0x7f6fa8115b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.566+0000 7f6faf800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fa81059b0 0x7f6fa81160c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.566+0000 7f6faf800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa8116710 con 0x7f6fa8105010 2024-04-16T05:38:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.566+0000 7f6faf800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa810fcf0 con 0x7f6fa81059b0 2024-04-16T05:38:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.567+0000 7f6face00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fa8105010 0x7f6fa8115b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.567+0000 7f6face00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fa8105010 0x7f6fa8115b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50620/0 (socket says 172.21.15.27:50620) 2024-04-16T05:38:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.567+0000 7f6face00700 1 -- 172.21.15.27:0/4052432350 learned_addr learned my addr 172.21.15.27:0/4052432350 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:38:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.567+0000 7f6fa7e00700 1 --2- 172.21.15.27:0/4052432350 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fa81059b0 0x7f6fa81160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.567+0000 7f6face00700 1 -- 172.21.15.27:0/4052432350 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fa81059b0 msgr2=0x7f6fa81160c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.567+0000 7f6face00700 1 --2- 172.21.15.27:0/4052432350 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fa81059b0 0x7f6fa81160c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.567+0000 7f6face00700 1 -- 172.21.15.27:0/4052432350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f94005040 con 0x7f6fa8105010 2024-04-16T05:38:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.568+0000 7f6face00700 1 --2- 172.21.15.27:0/4052432350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fa8105010 0x7f6fa8115b80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c00c8c0 tx=0x7f6f9c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:38:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.568+0000 7f6fa4c00700 1 -- 172.21.15.27:0/4052432350 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f9c00a980 con 0x7f6fa8105010 2024-04-16T05:38:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.568+0000 7f6fa4c00700 1 -- 172.21.15.27:0/4052432350 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f9c00ce60 con 0x7f6fa8105010 2024-04-16T05:38:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.568+0000 7f6fa4c00700 1 -- 172.21.15.27:0/4052432350 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f9c0115e0 con 0x7f6fa8105010 2024-04-16T05:38:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.568+0000 7f6faf800700 1 -- 172.21.15.27:0/4052432350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fa810ffd0 con 0x7f6fa8105010 2024-04-16T05:38:59.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.568+0000 7f6faf800700 1 -- 172.21.15.27:0/4052432350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fa8110590 con 0x7f6fa8105010 2024-04-16T05:38:59.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.569+0000 7f6faf800700 1 -- 172.21.15.27:0/4052432350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fa8100760 con 0x7f6fa8105010 2024-04-16T05:38:59.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.570+0000 7f6fa4c00700 1 -- 172.21.15.27:0/4052432350 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f6f9c012400 con 0x7f6fa8105010 2024-04-16T05:38:59.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.571+0000 7f6fa4c00700 1 --2- 172.21.15.27:0/4052432350 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f6f900692a0 0x7f6f9006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:38:59.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.572+0000 7f6fa4c00700 1 -- 172.21.15.27:0/4052432350 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6f9c086f20 con 0x7f6fa8105010 2024-04-16T05:38:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.572+0000 7f6fa7e00700 1 --2- 172.21.15.27:0/4052432350 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f6f900692a0 0x7f6f9006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:38:59.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.572+0000 7f6fa7e00700 1 --2- 172.21.15.27:0/4052432350 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f6f900692a0 0x7f6f9006b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8111420 tx=0x7f6f94009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:38:59.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.576+0000 7f6fa4c00700 1 -- 172.21.15.27:0/4052432350 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f6f9c057000 con 0x7f6fa8105010 2024-04-16T05:38:59.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.794+0000 7f6faf800700 1 -- 172.21.15.27:0/4052432350 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fa804c710 con 0x7f6fa8105010 2024-04-16T05:38:59.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.796+0000 7f6fa4c00700 1 -- 172.21.15.27:0/4052432350 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11084 (secure 0 0 0) 0x7f6f9c0126b0 con 0x7f6fa8105010 2024-04-16T05:38:59.795 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:38:59.795 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":40,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","created":"2024-04-16T05:31:01.142297+0000","modified":"2024-04-16T05:38:31.712493+0000","last_up_change":"2024-04-16T05:38:30.698288+0000","last_in_change":"2024-04-16T05:38:07.718747+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T05:36:52.435427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"4dd7e294-f625-4492-8bd3-18beab34b6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1557847192},{"type":"v1","addr":"172.21.15.27:6803","nonce":1557847192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1557847192},{"type":"v1","addr":"172.21.15.27:6805","nonce":1557847192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1557847192},{"type":"v1","addr":"172.21.15.27:6809","nonce":1557847192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1557847192},{"type":"v1","addr":"172.21.15.27:6807","nonce":1557847192}]},"public_addr":"172.21.15.27:6803/1557847192","cluster_addr":"172.21.15.27:6805/1557847192","heartbeat_back_addr":"172.21.15.27:6809/1557847192","heartbeat_front_addr":"172.21.15.27:6807/1557847192","state":["exists","up"]},{"osd":1,"uuid":"abec61aa-7a28-473b-b9d2-04e8d7de8a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2143934716},{"type":"v1","addr":"172.21.15.27:6811","nonce":2143934716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2143934716},{"type":"v1","addr":"172.21.15.27:6813","nonce":2143934716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2143934716},{"type":"v1","addr":"172.21.15.27:6817","nonce":2143934716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2143934716},{"type":"v1","addr":"172.21.15.27:6815","nonce":2143934716}]},"public_addr":"172.21.15.27:6811/2143934716","cluster_addr":"172.21.15.27:6813/2143934716","heartbeat_back_addr":"172.21.15.27:6817/2143934716","heartbeat_front_addr":"172.21.15.27:6815/2143934716","state":["exists","up"]},{"osd":2,"uuid":"e50778e4-bf1f-45eb-aeab-05e8dad8fa89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3370929967},{"type":"v1","addr":"172.21.15.27:6819","nonce":3370929967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3370929967},{"type":"v1","addr":"172.21.15.27:6821","nonce":3370929967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3370929967},{"type":"v1","addr":"172.21.15.27:6825","nonce":3370929967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3370929967},{"type":"v1","addr":"172.21.15.27:6823","nonce":3370929967}]},"public_addr":"172.21.15.27:6819/3370929967","cluster_addr":"172.21.15.27:6821/3370929967","heartbeat_back_addr":"172.21.15.27:6825/3370929967","heartbeat_front_addr":"172.21.15.27:6823/3370929967","state":["exists","up"]},{"osd":3,"uuid":"8f55dd70-a9d7-4bc1-a79c-028e66571137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":186542313},{"type":"v1","addr":"172.21.15.122:6801","nonce":186542313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":186542313},{"type":"v1","addr":"172.21.15.122:6803","nonce":186542313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":186542313},{"type":"v1","addr":"172.21.15.122:6807","nonce":186542313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":186542313},{"type":"v1","addr":"172.21.15.122:6805","nonce":186542313}]},"public_addr":"172.21.15.122:6801/186542313","cluster_addr":"172.21.15.122:6803/186542313","heartbeat_back_addr":"172.21.15.122:6807/186542313","heartbeat_front_addr":"172.21.15.122:6805/186542313","state":["exists","up"]},{"osd":4,"uuid":"ba8734db-cb7a-4242-b354-8ff1f524f25a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2655596639},{"type":"v1","addr":"172.21.15.122:6809","nonce":2655596639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2655596639},{"type":"v1","addr":"172.21.15.122:6811","nonce":2655596639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2655596639},{"type":"v1","addr":"172.21.15.122:6815","nonce":2655596639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2655596639},{"type":"v1","addr":"172.21.15.122:6813","nonce":2655596639}]},"public_addr":"172.21.15.122:6809/2655596639","cluster_addr":"172.21.15.122:6811/2655596639","heartbeat_back_addr":"172.21.15.122:6815/2655596639","heartbeat_front_addr":"172.21.15.122:6813/2655596639","state":["exists","up"]},{"osd":5,"uuid":"6e2753a7-d194-4103-b586-bfef8cbefa2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":3490413277},{"type":"v1","addr":"172.21.15.122:6817","nonce":3490413277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":3490413277},{"type":"v1","addr":"172.21.15.122:6819","nonce":3490413277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":3490413277},{"type":"v1","addr":"172.21.15.122:6823","nonce":3490413277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":3490413277},{"type":"v1","addr":"172.21.15.122:6821","nonce":3490413277}]},"public_addr":"172.21.15.122:6817/3490413277","cluster_addr":"172.21.15.122:6819/3490413277","heartbeat_back_addr":"172.21.15.122:6823/3490413277","heartbeat_front_addr":"172.21.15.122:6821/3490413277","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:35:28.179537+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:36:08.890364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:36:48.694394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:37:21.969883+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:37:55.840648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:38:29.596986+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/1860000032":"2024-04-17T05:33:49.465983+0000","172.21.15.27:0/602364106":"2024-04-17T05:33:49.465983+0000","172.21.15.27:6801/2700787222":"2024-04-17T05:33:49.465983+0000","172.21.15.27:0/3086153578":"2024-04-17T05:32:17.529651+0000","172.21.15.27:0/4258676557":"2024-04-17T05:32:17.529651+0000","172.21.15.27:0/2242461930":"2024-04-17T05:33:49.465983+0000","172.21.15.27:6801/2779081505":"2024-04-17T05:32:17.529651+0000","172.21.15.27:0/1435069943":"2024-04-17T05:31:34.994394+0000","172.21.15.27:6800/2700787222":"2024-04-17T05:33:49.465983+0000","172.21.15.27:6800/2173136120":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/3170976796":"2024-04-17T05:31:34.994394+0000","172.21.15.27:6801/2173136120":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/3713909498":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/3775595668":"2024-04-17T05:32:17.529651+0000","172.21.15.27:6800/2779081505":"2024-04-17T05:32:17.529651+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T05:38:59.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.800+0000 7f6faf800700 1 -- 172.21.15.27:0/4052432350 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f6f900692a0 msgr2=0x7f6f9006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:59.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.800+0000 7f6faf800700 1 --2- 172.21.15.27:0/4052432350 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f6f900692a0 0x7f6f9006b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8111420 tx=0x7f6f94009e30 comp rx=0 tx=0).stop 2024-04-16T05:38:59.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.800+0000 7f6faf800700 1 -- 172.21.15.27:0/4052432350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fa8105010 msgr2=0x7f6fa8115b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:38:59.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.800+0000 7f6faf800700 1 --2- 172.21.15.27:0/4052432350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fa8105010 0x7f6fa8115b80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c00c8c0 tx=0x7f6f9c0044c0 comp rx=0 tx=0).stop 2024-04-16T05:38:59.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.800+0000 7f6faf800700 1 -- 172.21.15.27:0/4052432350 shutdown_connections 2024-04-16T05:38:59.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.800+0000 7f6faf800700 1 --2- 172.21.15.27:0/4052432350 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f6f900692a0 0x7f6f9006b750 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:59.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.800+0000 7f6faf800700 1 --2- 172.21.15.27:0/4052432350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fa8105010 0x7f6fa8115b80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:59.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.800+0000 7f6faf800700 1 --2- 172.21.15.27:0/4052432350 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fa81059b0 0x7f6fa81160c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:38:59.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.800+0000 7f6faf800700 1 -- 172.21.15.27:0/4052432350 >> 172.21.15.27:0/4052432350 conn(0x7f6fa80faba0 msgr2=0x7f6fa80fbc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:38:59.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.800+0000 7f6faf800700 1 -- 172.21.15.27:0/4052432350 shutdown_connections 2024-04-16T05:38:59.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:38:59.801+0000 7f6faf800700 1 -- 172.21.15.27:0/4052432350 wait complete. 2024-04-16T05:39:00.628 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-16T05:39:00.628 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-16T05:39:00.629 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-16T05:39:00.630 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-16T05:39:00.630 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-16T05:39:00.631 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-16T05:39:01.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:00 smithi122 ceph-mon[35998]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:01.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:00 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/4052432350' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T05:39:01.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:00 smithi027 ceph-mon[29206]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:01.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:00 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/4052432350' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T05:39:01.230 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:39:01.235 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:39:01.334 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:39:01.633 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:39:02.189 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:39:02.967 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:39:03.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:02 smithi122 ceph-mon[35998]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:03.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:02 smithi027 ceph-mon[29206]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:04.710 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:04 smithi027 ceph-mon[29206]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:04.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:04 smithi122 ceph-mon[35998]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:05.644 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:05 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:39:05.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:05 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:39:06.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:06 smithi027 ceph-mon[29206]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:06.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:06 smithi122 ceph-mon[35998]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:07.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.922+0000 7fbf67000700 1 -- 172.21.15.27:0/2890809794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf600724f0 msgr2=0x7fbf600728c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:07.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.922+0000 7fbf67000700 1 --2- 172.21.15.27:0/2890809794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf600724f0 0x7fbf600728c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbf50004940 tx=0x7fbf50009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:07.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.922+0000 7fbf67000700 1 -- 172.21.15.27:0/2890809794 shutdown_connections 2024-04-16T05:39:07.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.922+0000 7fbf67000700 1 --2- 172.21.15.27:0/2890809794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbf60072e00 0x7fbf6010d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:07.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.922+0000 7fbf67000700 1 --2- 172.21.15.27:0/2890809794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf600724f0 0x7fbf600728c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:07.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.922+0000 7fbf67000700 1 -- 172.21.15.27:0/2890809794 >> 172.21.15.27:0/2890809794 conn(0x7fbf600faba0 msgr2=0x7fbf600fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:07.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.922+0000 7fbf67000700 1 -- 172.21.15.27:0/2890809794 shutdown_connections 2024-04-16T05:39:07.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.923+0000 7fbf67000700 1 -- 172.21.15.27:0/2890809794 wait complete. 2024-04-16T05:39:07.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.923+0000 7fbf67000700 1 Processor -- start 2024-04-16T05:39:07.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.924+0000 7fbf67000700 1 -- start start 2024-04-16T05:39:07.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.924+0000 7fbf67000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf600724f0 0x7fbf6011a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:07.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.924+0000 7fbf67000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbf60072e00 0x7fbf6011a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:07.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.924+0000 7fbf67000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6011ac80 con 0x7fbf600724f0 2024-04-16T05:39:07.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.924+0000 7fbf67000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf60114090 con 0x7fbf60072e00 2024-04-16T05:39:07.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.924+0000 7fbf5fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf600724f0 0x7fbf6011a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:07.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.924+0000 7fbf5fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf600724f0 0x7fbf6011a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58806/0 (socket says 172.21.15.27:58806) 2024-04-16T05:39:07.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.925+0000 7fbf5fe00700 1 -- 172.21.15.27:0/3672717765 learned_addr learned my addr 172.21.15.27:0/3672717765 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:39:07.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.925+0000 7fbf5f400700 1 --2- 172.21.15.27:0/3672717765 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbf60072e00 0x7fbf6011a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:07.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.925+0000 7fbf5fe00700 1 -- 172.21.15.27:0/3672717765 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbf60072e00 msgr2=0x7fbf6011a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:07.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.925+0000 7fbf5fe00700 1 --2- 172.21.15.27:0/3672717765 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbf60072e00 0x7fbf6011a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:07.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.925+0000 7fbf5fe00700 1 -- 172.21.15.27:0/3672717765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf50005040 con 0x7fbf600724f0 2024-04-16T05:39:07.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.925+0000 7fbf5fe00700 1 --2- 172.21.15.27:0/3672717765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf600724f0 0x7fbf6011a0f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbf50009180 tx=0x7fbf50009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:07.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.925+0000 7fbf57e00700 1 -- 172.21.15.27:0/3672717765 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf5000e040 con 0x7fbf600724f0 2024-04-16T05:39:07.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.925+0000 7fbf57e00700 1 -- 172.21.15.27:0/3672717765 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf50013420 con 0x7fbf600724f0 2024-04-16T05:39:07.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.925+0000 7fbf57e00700 1 -- 172.21.15.27:0/3672717765 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf5001c630 con 0x7fbf600724f0 2024-04-16T05:39:07.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.925+0000 7fbf67000700 1 -- 172.21.15.27:0/3672717765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf60114310 con 0x7fbf600724f0 2024-04-16T05:39:07.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.926+0000 7fbf67000700 1 -- 172.21.15.27:0/3672717765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf601148d0 con 0x7fbf600724f0 2024-04-16T05:39:07.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.926+0000 7fbf67000700 1 -- 172.21.15.27:0/3672717765 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbf60114eb0 con 0x7fbf600724f0 2024-04-16T05:39:07.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.928+0000 7fbf57e00700 1 -- 172.21.15.27:0/3672717765 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fbf50013590 con 0x7fbf600724f0 2024-04-16T05:39:07.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.929+0000 7fbf57e00700 1 --2- 172.21.15.27:0/3672717765 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fbf440692b0 0x7fbf4406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:07.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.929+0000 7fbf57e00700 1 -- 172.21.15.27:0/3672717765 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbf50088c60 con 0x7fbf600724f0 2024-04-16T05:39:07.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.929+0000 7fbf5f400700 1 --2- 172.21.15.27:0/3672717765 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fbf440692b0 0x7fbf4406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:07.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.929+0000 7fbf57e00700 1 --2- 172.21.15.27:0/3672717765 >> [v2:172.21.15.27:6810/2143934716,v1:172.21.15.27:6811/2143934716] conn(0x7fbf4406ef80 0x7fbf44071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:07.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.929+0000 7fbf57e00700 1 -- 172.21.15.27:0/3672717765 --> [v2:172.21.15.27:6810/2143934716,v1:172.21.15.27:6811/2143934716] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf44071a40 con 0x7fbf4406ef80 2024-04-16T05:39:07.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.929+0000 7fbf57e00700 1 -- 172.21.15.27:0/3672717765 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf50088fa0 con 0x7fbf600724f0 2024-04-16T05:39:07.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.929+0000 7fbf65000700 1 --2- 172.21.15.27:0/3672717765 >> [v2:172.21.15.27:6810/2143934716,v1:172.21.15.27:6811/2143934716] conn(0x7fbf4406ef80 0x7fbf44071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:07.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.930+0000 7fbf5f400700 1 --2- 172.21.15.27:0/3672717765 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fbf440692b0 0x7fbf4406b760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbf601159d0 tx=0x7fbf4c00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:07.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.930+0000 7fbf65000700 1 --2- 172.21.15.27:0/3672717765 >> [v2:172.21.15.27:6810/2143934716,v1:172.21.15.27:6811/2143934716] conn(0x7fbf4406ef80 0x7fbf44071390 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:07.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.931+0000 7fbf57e00700 1 -- 172.21.15.27:0/3672717765 <== osd.1 v2:172.21.15.27:6810/2143934716 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fbf44071a40 con 0x7fbf4406ef80 2024-04-16T05:39:07.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.947+0000 7fbf67000700 1 -- 172.21.15.27:0/3672717765 --> [v2:172.21.15.27:6810/2143934716,v1:172.21.15.27:6811/2143934716] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbf601144a0 con 0x7fbf4406ef80 2024-04-16T05:39:07.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.948+0000 7fbf57e00700 1 -- 172.21.15.27:0/3672717765 <== osd.1 v2:172.21.15.27:6810/2143934716 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbf601144a0 con 0x7fbf4406ef80 2024-04-16T05:39:07.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.949+0000 7fbf67000700 1 -- 172.21.15.27:0/3672717765 >> [v2:172.21.15.27:6810/2143934716,v1:172.21.15.27:6811/2143934716] conn(0x7fbf4406ef80 msgr2=0x7fbf44071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:07.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.949+0000 7fbf67000700 1 --2- 172.21.15.27:0/3672717765 >> [v2:172.21.15.27:6810/2143934716,v1:172.21.15.27:6811/2143934716] conn(0x7fbf4406ef80 0x7fbf44071390 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.949+0000 7fbf67000700 1 -- 172.21.15.27:0/3672717765 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fbf440692b0 msgr2=0x7fbf4406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.949+0000 7fbf67000700 1 --2- 172.21.15.27:0/3672717765 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fbf440692b0 0x7fbf4406b760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbf601159d0 tx=0x7fbf4c00b040 comp rx=0 tx=0).stop 2024-04-16T05:39:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.949+0000 7fbf67000700 1 -- 172.21.15.27:0/3672717765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf600724f0 msgr2=0x7fbf6011a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.949+0000 7fbf67000700 1 --2- 172.21.15.27:0/3672717765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf600724f0 0x7fbf6011a0f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbf50009180 tx=0x7fbf50009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:07.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.949+0000 7fbf67000700 1 -- 172.21.15.27:0/3672717765 shutdown_connections 2024-04-16T05:39:07.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.949+0000 7fbf67000700 1 --2- 172.21.15.27:0/3672717765 >> [v2:172.21.15.27:6810/2143934716,v1:172.21.15.27:6811/2143934716] conn(0x7fbf4406ef80 0x7fbf44071390 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:07.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.949+0000 7fbf67000700 1 --2- 172.21.15.27:0/3672717765 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fbf440692b0 0x7fbf4406b760 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:07.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.949+0000 7fbf67000700 1 --2- 172.21.15.27:0/3672717765 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbf600724f0 0x7fbf6011a0f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:07.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.949+0000 7fbf67000700 1 --2- 172.21.15.27:0/3672717765 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbf60072e00 0x7fbf6011a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:07.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.949+0000 7fbf67000700 1 -- 172.21.15.27:0/3672717765 >> 172.21.15.27:0/3672717765 conn(0x7fbf600faba0 msgr2=0x7fbf600faf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:07.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.949+0000 7fbf67000700 1 -- 172.21.15.27:0/3672717765 shutdown_connections 2024-04-16T05:39:07.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:07.949+0000 7fbf67000700 1 -- 172.21.15.27:0/3672717765 wait complete. 2024-04-16T05:39:08.873 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:08 smithi027 ceph-mon[29206]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:09.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:08 smithi122 ceph-mon[35998]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:09.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.288+0000 7f3d1da00700 1 -- 172.21.15.27:0/1584140860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d181037a0 msgr2=0x7f3d180fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:09.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.288+0000 7f3d1da00700 1 --2- 172.21.15.27:0/1584140860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d181037a0 0x7f3d180fce10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3d04004940 tx=0x7f3d04009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:09.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.288+0000 7f3d1da00700 1 -- 172.21.15.27:0/1584140860 shutdown_connections 2024-04-16T05:39:09.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.288+0000 7f3d1da00700 1 --2- 172.21.15.27:0/1584140860 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d181037a0 0x7f3d180fce10 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:09.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.288+0000 7f3d1da00700 1 --2- 172.21.15.27:0/1584140860 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d18102e00 0x7f3d181031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:09.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.288+0000 7f3d1da00700 1 -- 172.21.15.27:0/1584140860 >> 172.21.15.27:0/1584140860 conn(0x7f3d180f8950 msgr2=0x7f3d180fad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:09.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.289+0000 7f3d1da00700 1 -- 172.21.15.27:0/1584140860 shutdown_connections 2024-04-16T05:39:09.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.289+0000 7f3d1da00700 1 -- 172.21.15.27:0/1584140860 wait complete. 2024-04-16T05:39:09.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.290+0000 7f3d1da00700 1 Processor -- start 2024-04-16T05:39:09.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.290+0000 7f3d1da00700 1 -- start start 2024-04-16T05:39:09.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.290+0000 7f3d1da00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d18102e00 0x7f3d18115bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:09.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.290+0000 7f3d1da00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d181037a0 0x7f3d181160f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:09.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.290+0000 7f3d1da00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d18116740 con 0x7f3d181037a0 2024-04-16T05:39:09.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.290+0000 7f3d1da00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1810fd20 con 0x7f3d18102e00 2024-04-16T05:39:09.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.290+0000 7f3d16000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d181037a0 0x7f3d181160f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:09.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.291+0000 7f3d16000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d181037a0 0x7f3d181160f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58818/0 (socket says 172.21.15.27:58818) 2024-04-16T05:39:09.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.291+0000 7f3d16000700 1 -- 172.21.15.27:0/883488475 learned_addr learned my addr 172.21.15.27:0/883488475 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:39:09.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.291+0000 7f3d16a00700 1 --2- 172.21.15.27:0/883488475 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d18102e00 0x7f3d18115bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:09.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.291+0000 7f3d16000700 1 -- 172.21.15.27:0/883488475 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d18102e00 msgr2=0x7f3d18115bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:09.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.291+0000 7f3d16000700 1 --2- 172.21.15.27:0/883488475 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d18102e00 0x7f3d18115bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:09.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.291+0000 7f3d16000700 1 -- 172.21.15.27:0/883488475 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d04005040 con 0x7f3d181037a0 2024-04-16T05:39:09.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.291+0000 7f3d16000700 1 --2- 172.21.15.27:0/883488475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d181037a0 0x7f3d181160f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3d04009100 tx=0x7f3d04009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:09.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.292+0000 7f3d0ea00700 1 -- 172.21.15.27:0/883488475 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d0400e040 con 0x7f3d181037a0 2024-04-16T05:39:09.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.292+0000 7f3d0ea00700 1 -- 172.21.15.27:0/883488475 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d04013420 con 0x7f3d181037a0 2024-04-16T05:39:09.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.292+0000 7f3d1da00700 1 -- 172.21.15.27:0/883488475 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d1810ffa0 con 0x7f3d181037a0 2024-04-16T05:39:09.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.292+0000 7f3d0ea00700 1 -- 172.21.15.27:0/883488475 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d0401c630 con 0x7f3d181037a0 2024-04-16T05:39:09.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.292+0000 7f3d1da00700 1 -- 172.21.15.27:0/883488475 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d18110560 con 0x7f3d181037a0 2024-04-16T05:39:09.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.293+0000 7f3d1da00700 1 -- 172.21.15.27:0/883488475 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3d18110b00 con 0x7f3d181037a0 2024-04-16T05:39:09.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.294+0000 7f3d0ea00700 1 -- 172.21.15.27:0/883488475 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f3d04013590 con 0x7f3d181037a0 2024-04-16T05:39:09.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.295+0000 7f3d0ea00700 1 --2- 172.21.15.27:0/883488475 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f3cfc0692a0 0x7f3cfc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:09.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.295+0000 7f3d0ea00700 1 -- 172.21.15.27:0/883488475 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3d04088bd0 con 0x7f3d181037a0 2024-04-16T05:39:09.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.295+0000 7f3d16a00700 1 --2- 172.21.15.27:0/883488475 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f3cfc0692a0 0x7f3cfc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:09.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.295+0000 7f3d0ea00700 1 --2- 172.21.15.27:0/883488475 >> [v2:172.21.15.27:6802/1557847192,v1:172.21.15.27:6803/1557847192] conn(0x7f3cfc06ef70 0x7f3cfc071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:09.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.295+0000 7f3d0ea00700 1 -- 172.21.15.27:0/883488475 --> [v2:172.21.15.27:6802/1557847192,v1:172.21.15.27:6803/1557847192] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3cfc071a30 con 0x7f3cfc06ef70 2024-04-16T05:39:09.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.295+0000 7f3d0ea00700 1 -- 172.21.15.27:0/883488475 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d04088f10 con 0x7f3d181037a0 2024-04-16T05:39:09.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.296+0000 7f3d17400700 1 --2- 172.21.15.27:0/883488475 >> [v2:172.21.15.27:6802/1557847192,v1:172.21.15.27:6803/1557847192] conn(0x7f3cfc06ef70 0x7f3cfc071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:09.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.296+0000 7f3d16a00700 1 --2- 172.21.15.27:0/883488475 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f3cfc0692a0 0x7f3cfc06b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3d08007fd0 tx=0x7f3d08009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:09.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.296+0000 7f3d17400700 1 --2- 172.21.15.27:0/883488475 >> [v2:172.21.15.27:6802/1557847192,v1:172.21.15.27:6803/1557847192] conn(0x7f3cfc06ef70 0x7f3cfc071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:09.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.297+0000 7f3d0ea00700 1 -- 172.21.15.27:0/883488475 <== osd.0 v2:172.21.15.27:6802/1557847192 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f3cfc071a30 con 0x7f3cfc06ef70 2024-04-16T05:39:09.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.314+0000 7f3d1da00700 1 -- 172.21.15.27:0/883488475 --> [v2:172.21.15.27:6802/1557847192,v1:172.21.15.27:6803/1557847192] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3d1805e6c0 con 0x7f3cfc06ef70 2024-04-16T05:39:09.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.315+0000 7f3d0ea00700 1 -- 172.21.15.27:0/883488475 <== osd.0 v2:172.21.15.27:6802/1557847192 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3d1805e6c0 con 0x7f3cfc06ef70 2024-04-16T05:39:09.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.315+0000 7f3d1da00700 1 -- 172.21.15.27:0/883488475 >> [v2:172.21.15.27:6802/1557847192,v1:172.21.15.27:6803/1557847192] conn(0x7f3cfc06ef70 msgr2=0x7f3cfc071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:09.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.315+0000 7f3d1da00700 1 --2- 172.21.15.27:0/883488475 >> [v2:172.21.15.27:6802/1557847192,v1:172.21.15.27:6803/1557847192] conn(0x7f3cfc06ef70 0x7f3cfc071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:09.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.315+0000 7f3d1da00700 1 -- 172.21.15.27:0/883488475 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f3cfc0692a0 msgr2=0x7f3cfc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.315+0000 7f3d1da00700 1 --2- 172.21.15.27:0/883488475 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f3cfc0692a0 0x7f3cfc06b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3d08007fd0 tx=0x7f3d08009040 comp rx=0 tx=0).stop 2024-04-16T05:39:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.315+0000 7f3d1da00700 1 -- 172.21.15.27:0/883488475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d181037a0 msgr2=0x7f3d181160f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.315+0000 7f3d1da00700 1 --2- 172.21.15.27:0/883488475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d181037a0 0x7f3d181160f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3d04009100 tx=0x7f3d04009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.316+0000 7f3d1da00700 1 -- 172.21.15.27:0/883488475 shutdown_connections 2024-04-16T05:39:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.316+0000 7f3d1da00700 1 --2- 172.21.15.27:0/883488475 >> [v2:172.21.15.27:6802/1557847192,v1:172.21.15.27:6803/1557847192] conn(0x7f3cfc06ef70 0x7f3cfc071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.316+0000 7f3d1da00700 1 --2- 172.21.15.27:0/883488475 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f3cfc0692a0 0x7f3cfc06b750 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.316+0000 7f3d1da00700 1 --2- 172.21.15.27:0/883488475 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d18102e00 0x7f3d18115bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.316+0000 7f3d1da00700 1 --2- 172.21.15.27:0/883488475 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d181037a0 0x7f3d181160f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.316+0000 7f3d1da00700 1 -- 172.21.15.27:0/883488475 >> 172.21.15.27:0/883488475 conn(0x7f3d180f8950 msgr2=0x7f3d181021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.316+0000 7f3d1da00700 1 -- 172.21.15.27:0/883488475 shutdown_connections 2024-04-16T05:39:09.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:09.316+0000 7f3d1da00700 1 -- 172.21.15.27:0/883488475 wait complete. 2024-04-16T05:39:10.329 INFO:teuthology.orchestra.run.smithi027.stdout:60129542182 2024-04-16T05:39:10.329 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-16T05:39:10.824 INFO:teuthology.orchestra.run.smithi027.stdout:38654705709 2024-04-16T05:39:10.825 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-16T05:39:10.878 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:10 smithi027 ceph-mon[29206]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:11.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:10 smithi122 ceph-mon[35998]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:12.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.090+0000 7fb649000700 1 -- 172.21.15.27:0/327500268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6441059b0 msgr2=0x7fb6440ff020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:12.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.090+0000 7fb649000700 1 --2- 172.21.15.27:0/327500268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6441059b0 0x7fb6440ff020 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb630004940 tx=0x7fb630009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:12.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.090+0000 7fb649000700 1 -- 172.21.15.27:0/327500268 shutdown_connections 2024-04-16T05:39:12.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.090+0000 7fb649000700 1 --2- 172.21.15.27:0/327500268 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6441059b0 0x7fb6440ff020 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:12.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.090+0000 7fb649000700 1 --2- 172.21.15.27:0/327500268 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb644105010 0x7fb6441053e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:12.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.090+0000 7fb649000700 1 -- 172.21.15.27:0/327500268 >> 172.21.15.27:0/327500268 conn(0x7fb6440faba0 msgr2=0x7fb6440fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:12.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.090+0000 7fb649000700 1 -- 172.21.15.27:0/327500268 shutdown_connections 2024-04-16T05:39:12.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.090+0000 7fb649000700 1 -- 172.21.15.27:0/327500268 wait complete. 2024-04-16T05:39:12.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.091+0000 7fb649000700 1 Processor -- start 2024-04-16T05:39:12.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.091+0000 7fb649000700 1 -- start start 2024-04-16T05:39:12.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.092+0000 7fb649000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb644105010 0x7fb644119080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:12.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.092+0000 7fb649000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6441059b0 0x7fb644119600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:12.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.092+0000 7fb649000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb644119c50 con 0x7fb6441059b0 2024-04-16T05:39:12.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.092+0000 7fb649000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb644074ec0 con 0x7fb644105010 2024-04-16T05:39:12.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.092+0000 7fb641400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6441059b0 0x7fb644119600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.092+0000 7fb641400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6441059b0 0x7fb644119600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58834/0 (socket says 172.21.15.27:58834) 2024-04-16T05:39:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.092+0000 7fb641400700 1 -- 172.21.15.27:0/3743221022 learned_addr learned my addr 172.21.15.27:0/3743221022 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:39:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.092+0000 7fb641e00700 1 --2- 172.21.15.27:0/3743221022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb644105010 0x7fb644119080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.093+0000 7fb641400700 1 -- 172.21.15.27:0/3743221022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb644105010 msgr2=0x7fb644119080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.093+0000 7fb641400700 1 --2- 172.21.15.27:0/3743221022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb644105010 0x7fb644119080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.093+0000 7fb641400700 1 -- 172.21.15.27:0/3743221022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb630005040 con 0x7fb6441059b0 2024-04-16T05:39:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.093+0000 7fb641400700 1 --2- 172.21.15.27:0/3743221022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6441059b0 0x7fb644119600 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb630009180 tx=0x7fb630009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.093+0000 7fb63a000700 1 -- 172.21.15.27:0/3743221022 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb63000e040 con 0x7fb6441059b0 2024-04-16T05:39:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.093+0000 7fb63a000700 1 -- 172.21.15.27:0/3743221022 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb63000bbc0 con 0x7fb6441059b0 2024-04-16T05:39:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.093+0000 7fb63a000700 1 -- 172.21.15.27:0/3743221022 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb63001c3f0 con 0x7fb6441059b0 2024-04-16T05:39:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.093+0000 7fb649000700 1 -- 172.21.15.27:0/3743221022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6440750e0 con 0x7fb6441059b0 2024-04-16T05:39:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.094+0000 7fb649000700 1 -- 172.21.15.27:0/3743221022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb644075620 con 0x7fb6441059b0 2024-04-16T05:39:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.095+0000 7fb649000700 1 -- 172.21.15.27:0/3743221022 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb644075a80 con 0x7fb6441059b0 2024-04-16T05:39:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.096+0000 7fb63a000700 1 -- 172.21.15.27:0/3743221022 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fb63000bd30 con 0x7fb6441059b0 2024-04-16T05:39:12.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.097+0000 7fb63a000700 1 --2- 172.21.15.27:0/3743221022 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb6280692a0 0x7fb62806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.097+0000 7fb63a000700 1 -- 172.21.15.27:0/3743221022 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb630088cc0 con 0x7fb6441059b0 2024-04-16T05:39:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.097+0000 7fb63a000700 1 --2- 172.21.15.27:0/3743221022 >> [v2:172.21.15.27:6818/3370929967,v1:172.21.15.27:6819/3370929967] conn(0x7fb62806ef70 0x7fb628071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.097+0000 7fb641e00700 1 --2- 172.21.15.27:0/3743221022 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb6280692a0 0x7fb62806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.097+0000 7fb63a000700 1 -- 172.21.15.27:0/3743221022 --> [v2:172.21.15.27:6818/3370929967,v1:172.21.15.27:6819/3370929967] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb628071a30 con 0x7fb62806ef70 2024-04-16T05:39:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.097+0000 7fb63a000700 1 -- 172.21.15.27:0/3743221022 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb630089000 con 0x7fb6441059b0 2024-04-16T05:39:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.098+0000 7fb642800700 1 --2- 172.21.15.27:0/3743221022 >> [v2:172.21.15.27:6818/3370929967,v1:172.21.15.27:6819/3370929967] conn(0x7fb62806ef70 0x7fb628071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.098+0000 7fb641e00700 1 --2- 172.21.15.27:0/3743221022 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb6280692a0 0x7fb62806b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb634007fd0 tx=0x7fb634009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.098+0000 7fb642800700 1 --2- 172.21.15.27:0/3743221022 >> [v2:172.21.15.27:6818/3370929967,v1:172.21.15.27:6819/3370929967] conn(0x7fb62806ef70 0x7fb628071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.099+0000 7fb63a000700 1 -- 172.21.15.27:0/3743221022 <== osd.2 v2:172.21.15.27:6818/3370929967 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fb628071a30 con 0x7fb62806ef70 2024-04-16T05:39:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.121+0000 7fb649000700 1 -- 172.21.15.27:0/3743221022 --> [v2:172.21.15.27:6818/3370929967,v1:172.21.15.27:6819/3370929967] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb644100760 con 0x7fb62806ef70 2024-04-16T05:39:12.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.121+0000 7fb63a000700 1 -- 172.21.15.27:0/3743221022 <== osd.2 v2:172.21.15.27:6818/3370929967 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb644100760 con 0x7fb62806ef70 2024-04-16T05:39:12.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.122+0000 7fb649000700 1 -- 172.21.15.27:0/3743221022 >> [v2:172.21.15.27:6818/3370929967,v1:172.21.15.27:6819/3370929967] conn(0x7fb62806ef70 msgr2=0x7fb628071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:12.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.122+0000 7fb649000700 1 --2- 172.21.15.27:0/3743221022 >> [v2:172.21.15.27:6818/3370929967,v1:172.21.15.27:6819/3370929967] conn(0x7fb62806ef70 0x7fb628071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:12.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.122+0000 7fb649000700 1 -- 172.21.15.27:0/3743221022 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb6280692a0 msgr2=0x7fb62806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:12.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.122+0000 7fb649000700 1 --2- 172.21.15.27:0/3743221022 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb6280692a0 0x7fb62806b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb634007fd0 tx=0x7fb634009040 comp rx=0 tx=0).stop 2024-04-16T05:39:12.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.122+0000 7fb649000700 1 -- 172.21.15.27:0/3743221022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6441059b0 msgr2=0x7fb644119600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:12.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.122+0000 7fb649000700 1 --2- 172.21.15.27:0/3743221022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6441059b0 0x7fb644119600 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb630009180 tx=0x7fb630009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:12.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.122+0000 7fb649000700 1 -- 172.21.15.27:0/3743221022 shutdown_connections 2024-04-16T05:39:12.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.122+0000 7fb649000700 1 --2- 172.21.15.27:0/3743221022 >> [v2:172.21.15.27:6818/3370929967,v1:172.21.15.27:6819/3370929967] conn(0x7fb62806ef70 0x7fb628071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:12.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.122+0000 7fb649000700 1 --2- 172.21.15.27:0/3743221022 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb6280692a0 0x7fb62806b750 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:12.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.122+0000 7fb649000700 1 --2- 172.21.15.27:0/3743221022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb644105010 0x7fb644119080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:12.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.122+0000 7fb649000700 1 --2- 172.21.15.27:0/3743221022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6441059b0 0x7fb644119600 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:12.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.122+0000 7fb649000700 1 -- 172.21.15.27:0/3743221022 >> 172.21.15.27:0/3743221022 conn(0x7fb6440faba0 msgr2=0x7fb6440fbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:12.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.123+0000 7fb649000700 1 -- 172.21.15.27:0/3743221022 shutdown_connections 2024-04-16T05:39:12.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.123+0000 7fb649000700 1 -- 172.21.15.27:0/3743221022 wait complete. 2024-04-16T05:39:12.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.395+0000 7fdd11a00700 1 -- 172.21.15.27:0/4153931379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd0c105020 msgr2=0x7fdd0c1053f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:12.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.395+0000 7fdd11a00700 1 --2- 172.21.15.27:0/4153931379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd0c105020 0x7fdd0c1053f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc004940 tx=0x7fdcfc009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:12.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.395+0000 7fdd11a00700 1 -- 172.21.15.27:0/4153931379 shutdown_connections 2024-04-16T05:39:12.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.395+0000 7fdd11a00700 1 --2- 172.21.15.27:0/4153931379 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd0c1059c0 0x7fdd0c0ff030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:12.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.395+0000 7fdd11a00700 1 --2- 172.21.15.27:0/4153931379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd0c105020 0x7fdd0c1053f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:12.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.395+0000 7fdd11a00700 1 -- 172.21.15.27:0/4153931379 >> 172.21.15.27:0/4153931379 conn(0x7fdd0c0fabd0 msgr2=0x7fdd0c0fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:12.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.396+0000 7fdd11a00700 1 -- 172.21.15.27:0/4153931379 shutdown_connections 2024-04-16T05:39:12.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.396+0000 7fdd11a00700 1 -- 172.21.15.27:0/4153931379 wait complete. 2024-04-16T05:39:12.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.397+0000 7fdd11a00700 1 Processor -- start 2024-04-16T05:39:12.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.397+0000 7fdd11a00700 1 -- start start 2024-04-16T05:39:12.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.398+0000 7fdd11a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd0c105020 0x7fdd0c115d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:12.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.398+0000 7fdd11a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd0c1059c0 0x7fdd0c116270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:12.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.398+0000 7fdd11a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c1168c0 con 0x7fdd0c105020 2024-04-16T05:39:12.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.398+0000 7fdd11a00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c10fc90 con 0x7fdd0c1059c0 2024-04-16T05:39:12.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.398+0000 7fdd0aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd0c105020 0x7fdd0c115d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:12.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.398+0000 7fdd0aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd0c105020 0x7fdd0c115d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58846/0 (socket says 172.21.15.27:58846) 2024-04-16T05:39:12.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.398+0000 7fdd0aa00700 1 -- 172.21.15.27:0/3208647740 learned_addr learned my addr 172.21.15.27:0/3208647740 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:39:12.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.398+0000 7fdd0a000700 1 --2- 172.21.15.27:0/3208647740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd0c1059c0 0x7fdd0c116270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:12.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.398+0000 7fdd0aa00700 1 -- 172.21.15.27:0/3208647740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd0c1059c0 msgr2=0x7fdd0c116270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:12.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.398+0000 7fdd0aa00700 1 --2- 172.21.15.27:0/3208647740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd0c1059c0 0x7fdd0c116270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:12.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.398+0000 7fdd0aa00700 1 -- 172.21.15.27:0/3208647740 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcfc005040 con 0x7fdd0c105020 2024-04-16T05:39:12.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.399+0000 7fdd0aa00700 1 --2- 172.21.15.27:0/3208647740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd0c105020 0x7fdd0c115d30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc009b30 tx=0x7fdcfc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:12.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.399+0000 7fdd02a00700 1 -- 172.21.15.27:0/3208647740 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcfc00e070 con 0x7fdd0c105020 2024-04-16T05:39:12.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.399+0000 7fdd02a00700 1 -- 172.21.15.27:0/3208647740 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdcfc013cd0 con 0x7fdd0c105020 2024-04-16T05:39:12.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.399+0000 7fdd11a00700 1 -- 172.21.15.27:0/3208647740 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd0c10ff10 con 0x7fdd0c105020 2024-04-16T05:39:12.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.399+0000 7fdd11a00700 1 -- 172.21.15.27:0/3208647740 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd0c1103f0 con 0x7fdd0c105020 2024-04-16T05:39:12.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.399+0000 7fdd02a00700 1 -- 172.21.15.27:0/3208647740 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcfc00b720 con 0x7fdd0c105020 2024-04-16T05:39:12.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.400+0000 7fdd11a00700 1 -- 172.21.15.27:0/3208647740 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdcec000fd0 con 0x7fdd0c105020 2024-04-16T05:39:12.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.401+0000 7fdd02a00700 1 -- 172.21.15.27:0/3208647740 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fdcfc01c400 con 0x7fdd0c105020 2024-04-16T05:39:12.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.402+0000 7fdd02a00700 1 --2- 172.21.15.27:0/3208647740 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fdcf00692a0 0x7fdcf006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.402+0000 7fdd02a00700 1 -- 172.21.15.27:0/3208647740 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdcfc088790 con 0x7fdd0c105020 2024-04-16T05:39:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.403+0000 7fdd02a00700 1 --2- 172.21.15.27:0/3208647740 >> [v2:172.21.15.122:6808/2655596639,v1:172.21.15.122:6809/2655596639] conn(0x7fdcf006ef70 0x7fdcf0071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.403+0000 7fdd0a000700 1 --2- 172.21.15.27:0/3208647740 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fdcf00692a0 0x7fdcf006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.403+0000 7fdd02a00700 1 -- 172.21.15.27:0/3208647740 --> [v2:172.21.15.122:6808/2655596639,v1:172.21.15.122:6809/2655596639] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdcf0071a30 con 0x7fdcf006ef70 2024-04-16T05:39:12.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.403+0000 7fdd02a00700 1 -- 172.21.15.27:0/3208647740 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdcfc08d220 con 0x7fdd0c105020 2024-04-16T05:39:12.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.403+0000 7fdd0a000700 1 --2- 172.21.15.27:0/3208647740 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fdcf00692a0 0x7fdcf006b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8004600 tx=0x7fdcf800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:12.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.403+0000 7fdd0b400700 1 --2- 172.21.15.27:0/3208647740 >> [v2:172.21.15.122:6808/2655596639,v1:172.21.15.122:6809/2655596639] conn(0x7fdcf006ef70 0x7fdcf0071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:12.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.404+0000 7fdd0b400700 1 --2- 172.21.15.27:0/3208647740 >> [v2:172.21.15.122:6808/2655596639,v1:172.21.15.122:6809/2655596639] conn(0x7fdcf006ef70 0x7fdcf0071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:12.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.407+0000 7fdd02a00700 1 -- 172.21.15.27:0/3208647740 <== osd.4 v2:172.21.15.122:6808/2655596639 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fdcf0071a30 con 0x7fdcf006ef70 2024-04-16T05:39:12.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.423+0000 7fdd11a00700 1 -- 172.21.15.27:0/3208647740 --> [v2:172.21.15.122:6808/2655596639,v1:172.21.15.122:6809/2655596639] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdcec002d30 con 0x7fdcf006ef70 2024-04-16T05:39:12.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.424+0000 7fdd02a00700 1 -- 172.21.15.27:0/3208647740 <== osd.4 v2:172.21.15.122:6808/2655596639 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fdcec002d30 con 0x7fdcf006ef70 2024-04-16T05:39:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.425+0000 7fdd11a00700 1 -- 172.21.15.27:0/3208647740 >> [v2:172.21.15.122:6808/2655596639,v1:172.21.15.122:6809/2655596639] conn(0x7fdcf006ef70 msgr2=0x7fdcf0071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.425+0000 7fdd11a00700 1 --2- 172.21.15.27:0/3208647740 >> [v2:172.21.15.122:6808/2655596639,v1:172.21.15.122:6809/2655596639] conn(0x7fdcf006ef70 0x7fdcf0071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.425+0000 7fdd11a00700 1 -- 172.21.15.27:0/3208647740 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fdcf00692a0 msgr2=0x7fdcf006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.425+0000 7fdd11a00700 1 --2- 172.21.15.27:0/3208647740 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fdcf00692a0 0x7fdcf006b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8004600 tx=0x7fdcf800b040 comp rx=0 tx=0).stop 2024-04-16T05:39:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.425+0000 7fdd11a00700 1 -- 172.21.15.27:0/3208647740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd0c105020 msgr2=0x7fdd0c115d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.425+0000 7fdd11a00700 1 --2- 172.21.15.27:0/3208647740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd0c105020 0x7fdd0c115d30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc009b30 tx=0x7fdcfc009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:12.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.425+0000 7fdd11a00700 1 -- 172.21.15.27:0/3208647740 shutdown_connections 2024-04-16T05:39:12.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.425+0000 7fdd11a00700 1 --2- 172.21.15.27:0/3208647740 >> [v2:172.21.15.122:6808/2655596639,v1:172.21.15.122:6809/2655596639] conn(0x7fdcf006ef70 0x7fdcf0071380 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:12.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.425+0000 7fdd11a00700 1 --2- 172.21.15.27:0/3208647740 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fdcf00692a0 0x7fdcf006b750 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:12.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.425+0000 7fdd11a00700 1 --2- 172.21.15.27:0/3208647740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd0c105020 0x7fdd0c115d30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:12.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.425+0000 7fdd11a00700 1 --2- 172.21.15.27:0/3208647740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd0c1059c0 0x7fdd0c116270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:12.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.425+0000 7fdd11a00700 1 -- 172.21.15.27:0/3208647740 >> 172.21.15.27:0/3208647740 conn(0x7fdd0c0fabd0 msgr2=0x7fdd0c0fbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:12.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.426+0000 7fdd11a00700 1 -- 172.21.15.27:0/3208647740 shutdown_connections 2024-04-16T05:39:12.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:12.426+0000 7fdd11a00700 1 -- 172.21.15.27:0/3208647740 wait complete. 2024-04-16T05:39:12.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:12 smithi027 ceph-mon[29206]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:13.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:12 smithi122 ceph-mon[35998]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:13.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.430+0000 7f051fe00700 1 -- 172.21.15.27:0/2212899224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0520109280 msgr2=0x7f0520109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:13.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.430+0000 7f051fe00700 1 --2- 172.21.15.27:0/2212899224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0520109280 0x7f0520109650 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0510004940 tx=0x7f0510009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:13.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.430+0000 7f051fe00700 1 -- 172.21.15.27:0/2212899224 shutdown_connections 2024-04-16T05:39:13.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.430+0000 7f051fe00700 1 --2- 172.21.15.27:0/2212899224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0520109c20 0x7f0520103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:13.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.430+0000 7f051fe00700 1 --2- 172.21.15.27:0/2212899224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0520109280 0x7f0520109650 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:13.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.430+0000 7f051fe00700 1 -- 172.21.15.27:0/2212899224 >> 172.21.15.27:0/2212899224 conn(0x7f0520072a60 msgr2=0x7f0520074e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:13.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.430+0000 7f051fe00700 1 -- 172.21.15.27:0/2212899224 shutdown_connections 2024-04-16T05:39:13.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.430+0000 7f051fe00700 1 -- 172.21.15.27:0/2212899224 wait complete. 2024-04-16T05:39:13.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.431+0000 7f051fe00700 1 Processor -- start 2024-04-16T05:39:13.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.431+0000 7f051fe00700 1 -- start start 2024-04-16T05:39:13.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.431+0000 7f051fe00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0520109280 0x7f052011a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:13.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.431+0000 7f051fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0520109c20 0x7f052011a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:13.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.431+0000 7f051fe00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052011ac20 con 0x7f0520109c20 2024-04-16T05:39:13.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.431+0000 7f051fe00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0520113f60 con 0x7f0520109280 2024-04-16T05:39:13.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.432+0000 7f051e000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0520109c20 0x7f052011a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:13.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.432+0000 7f051e000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0520109c20 0x7f052011a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58860/0 (socket says 172.21.15.27:58860) 2024-04-16T05:39:13.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.432+0000 7f051e000700 1 -- 172.21.15.27:0/1244693947 learned_addr learned my addr 172.21.15.27:0/1244693947 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:39:13.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.432+0000 7f051ea00700 1 --2- 172.21.15.27:0/1244693947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0520109280 0x7f052011a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:13.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.432+0000 7f051e000700 1 -- 172.21.15.27:0/1244693947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0520109280 msgr2=0x7f052011a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:13.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.432+0000 7f051e000700 1 --2- 172.21.15.27:0/1244693947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0520109280 0x7f052011a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:13.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.432+0000 7f051e000700 1 -- 172.21.15.27:0/1244693947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0510005040 con 0x7f0520109c20 2024-04-16T05:39:13.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.433+0000 7f051e000700 1 --2- 172.21.15.27:0/1244693947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0520109c20 0x7f052011a540 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f050c00b790 tx=0x7f050c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:13.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.433+0000 7f0516a00700 1 -- 172.21.15.27:0/1244693947 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f050c00f6f0 con 0x7f0520109c20 2024-04-16T05:39:13.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.433+0000 7f0516a00700 1 -- 172.21.15.27:0/1244693947 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f050c00fd20 con 0x7f0520109c20 2024-04-16T05:39:13.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.433+0000 7f0516a00700 1 -- 172.21.15.27:0/1244693947 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f050c0103a0 con 0x7f0520109c20 2024-04-16T05:39:13.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.433+0000 7f051fe00700 1 -- 172.21.15.27:0/1244693947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05201141e0 con 0x7f0520109c20 2024-04-16T05:39:13.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.433+0000 7f051fe00700 1 -- 172.21.15.27:0/1244693947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05201147a0 con 0x7f0520109c20 2024-04-16T05:39:13.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.434+0000 7f051fe00700 1 -- 172.21.15.27:0/1244693947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0520114d30 con 0x7f0520109c20 2024-04-16T05:39:13.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.435+0000 7f0516a00700 1 -- 172.21.15.27:0/1244693947 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f050c011400 con 0x7f0520109c20 2024-04-16T05:39:13.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.436+0000 7f0516a00700 1 --2- 172.21.15.27:0/1244693947 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f05040692a0 0x7f050406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:13.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.436+0000 7f0516a00700 1 -- 172.21.15.27:0/1244693947 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f050c086c50 con 0x7f0520109c20 2024-04-16T05:39:13.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.436+0000 7f051ea00700 1 --2- 172.21.15.27:0/1244693947 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f05040692a0 0x7f050406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:13.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.437+0000 7f0516a00700 1 --2- 172.21.15.27:0/1244693947 >> [v2:172.21.15.122:6816/3490413277,v1:172.21.15.122:6817/3490413277] conn(0x7f050406ef70 0x7f0504071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:13.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.437+0000 7f0516a00700 1 -- 172.21.15.27:0/1244693947 --> [v2:172.21.15.122:6816/3490413277,v1:172.21.15.122:6817/3490413277] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0504071a30 con 0x7f050406ef70 2024-04-16T05:39:13.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.437+0000 7f0516a00700 1 -- 172.21.15.27:0/1244693947 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f050c08c250 con 0x7f0520109c20 2024-04-16T05:39:13.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.437+0000 7f051ea00700 1 --2- 172.21.15.27:0/1244693947 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f05040692a0 0x7f050406b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f051000b3b0 tx=0x7f0510009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:13.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.437+0000 7f051f400700 1 --2- 172.21.15.27:0/1244693947 >> [v2:172.21.15.122:6816/3490413277,v1:172.21.15.122:6817/3490413277] conn(0x7f050406ef70 0x7f0504071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:13.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.438+0000 7f051f400700 1 --2- 172.21.15.27:0/1244693947 >> [v2:172.21.15.122:6816/3490413277,v1:172.21.15.122:6817/3490413277] conn(0x7f050406ef70 0x7f0504071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:13.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.440+0000 7f0516a00700 1 -- 172.21.15.27:0/1244693947 <== osd.5 v2:172.21.15.122:6816/3490413277 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f0504071a30 con 0x7f050406ef70 2024-04-16T05:39:13.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.457+0000 7f051fe00700 1 -- 172.21.15.27:0/1244693947 --> [v2:172.21.15.122:6816/3490413277,v1:172.21.15.122:6817/3490413277] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f05200752f0 con 0x7f050406ef70 2024-04-16T05:39:13.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.458+0000 7f0516a00700 1 -- 172.21.15.27:0/1244693947 <== osd.5 v2:172.21.15.122:6816/3490413277 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f05200752f0 con 0x7f050406ef70 2024-04-16T05:39:13.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.458+0000 7f051fe00700 1 -- 172.21.15.27:0/1244693947 >> [v2:172.21.15.122:6816/3490413277,v1:172.21.15.122:6817/3490413277] conn(0x7f050406ef70 msgr2=0x7f0504071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:13.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.458+0000 7f051fe00700 1 --2- 172.21.15.27:0/1244693947 >> [v2:172.21.15.122:6816/3490413277,v1:172.21.15.122:6817/3490413277] conn(0x7f050406ef70 0x7f0504071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:13.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.458+0000 7f051fe00700 1 -- 172.21.15.27:0/1244693947 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f05040692a0 msgr2=0x7f050406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:13.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.458+0000 7f051fe00700 1 --2- 172.21.15.27:0/1244693947 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f05040692a0 0x7f050406b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f051000b3b0 tx=0x7f0510009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.458+0000 7f051fe00700 1 -- 172.21.15.27:0/1244693947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0520109c20 msgr2=0x7f052011a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.458+0000 7f051fe00700 1 --2- 172.21.15.27:0/1244693947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0520109c20 0x7f052011a540 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f050c00b790 tx=0x7f050c009220 comp rx=0 tx=0).stop 2024-04-16T05:39:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.459+0000 7f051fe00700 1 -- 172.21.15.27:0/1244693947 shutdown_connections 2024-04-16T05:39:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.459+0000 7f051fe00700 1 --2- 172.21.15.27:0/1244693947 >> [v2:172.21.15.122:6816/3490413277,v1:172.21.15.122:6817/3490413277] conn(0x7f050406ef70 0x7f0504071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.459+0000 7f051fe00700 1 --2- 172.21.15.27:0/1244693947 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f05040692a0 0x7f050406b750 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.459+0000 7f051fe00700 1 --2- 172.21.15.27:0/1244693947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0520109280 0x7f052011a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.459+0000 7f051fe00700 1 --2- 172.21.15.27:0/1244693947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0520109c20 0x7f052011a540 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.459+0000 7f051fe00700 1 -- 172.21.15.27:0/1244693947 >> 172.21.15.27:0/1244693947 conn(0x7f0520072a60 msgr2=0x7f05200749a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.459+0000 7f051fe00700 1 -- 172.21.15.27:0/1244693947 shutdown_connections 2024-04-16T05:39:13.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:13.459+0000 7f051fe00700 1 -- 172.21.15.27:0/1244693947 wait complete. 2024-04-16T05:39:14.496 INFO:teuthology.orchestra.run.smithi027.stdout:81604378655 2024-04-16T05:39:14.497 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-16T05:39:14.606 INFO:teuthology.orchestra.run.smithi027.stdout:167503724554 2024-04-16T05:39:14.606 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-16T05:39:14.695 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:14 smithi027 ceph-mon[29206]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:14.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.834+0000 7fb681400700 1 -- 172.21.15.27:0/2771880197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb67c102e10 msgr2=0x7fb67c1031e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:14.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.834+0000 7fb681400700 1 --2- 172.21.15.27:0/2771880197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb67c102e10 0x7fb67c1031e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb66c004940 tx=0x7fb66c009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:14.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.834+0000 7fb681400700 1 -- 172.21.15.27:0/2771880197 shutdown_connections 2024-04-16T05:39:14.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.834+0000 7fb681400700 1 --2- 172.21.15.27:0/2771880197 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb67c1037b0 0x7fb67c0fce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:14.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.834+0000 7fb681400700 1 --2- 172.21.15.27:0/2771880197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb67c102e10 0x7fb67c1031e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:14.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.834+0000 7fb681400700 1 -- 172.21.15.27:0/2771880197 >> 172.21.15.27:0/2771880197 conn(0x7fb67c0f89a0 msgr2=0x7fb67c0fadb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:14.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.835+0000 7fb681400700 1 -- 172.21.15.27:0/2771880197 shutdown_connections 2024-04-16T05:39:14.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.835+0000 7fb681400700 1 -- 172.21.15.27:0/2771880197 wait complete. 2024-04-16T05:39:14.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.836+0000 7fb681400700 1 Processor -- start 2024-04-16T05:39:14.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.836+0000 7fb681400700 1 -- start start 2024-04-16T05:39:14.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.837+0000 7fb681400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb67c102e10 0x7fb67c115d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:14.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.837+0000 7fb681400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb67c1037b0 0x7fb67c1162d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:14.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.837+0000 7fb681400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c116920 con 0x7fb67c1037b0 2024-04-16T05:39:14.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.837+0000 7fb681400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c10fcf0 con 0x7fb67c102e10 2024-04-16T05:39:14.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.837+0000 7fb679e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb67c1037b0 0x7fb67c1162d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:14.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.837+0000 7fb679e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb67c1037b0 0x7fb67c1162d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49706/0 (socket says 172.21.15.27:49706) 2024-04-16T05:39:14.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.837+0000 7fb679e00700 1 -- 172.21.15.27:0/157627906 learned_addr learned my addr 172.21.15.27:0/157627906 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:39:14.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.838+0000 7fb67a800700 1 --2- 172.21.15.27:0/157627906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb67c102e10 0x7fb67c115d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:14.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.838+0000 7fb679e00700 1 -- 172.21.15.27:0/157627906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb67c102e10 msgr2=0x7fb67c115d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:14.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.838+0000 7fb679e00700 1 --2- 172.21.15.27:0/157627906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb67c102e10 0x7fb67c115d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:14.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.838+0000 7fb679e00700 1 -- 172.21.15.27:0/157627906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66c005040 con 0x7fb67c1037b0 2024-04-16T05:39:14.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.838+0000 7fb679e00700 1 --2- 172.21.15.27:0/157627906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb67c1037b0 0x7fb67c1162d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb66800c800 tx=0x7fb668004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:14.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.839+0000 7fb672a00700 1 -- 172.21.15.27:0/157627906 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6680106f0 con 0x7fb67c1037b0 2024-04-16T05:39:14.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.839+0000 7fb681400700 1 -- 172.21.15.27:0/157627906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb67c10ffd0 con 0x7fb67c1037b0 2024-04-16T05:39:14.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.839+0000 7fb681400700 1 -- 172.21.15.27:0/157627906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67c110590 con 0x7fb67c1037b0 2024-04-16T05:39:14.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.839+0000 7fb672a00700 1 -- 172.21.15.27:0/157627906 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb668010d20 con 0x7fb67c1037b0 2024-04-16T05:39:14.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.839+0000 7fb672a00700 1 -- 172.21.15.27:0/157627906 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6680113a0 con 0x7fb67c1037b0 2024-04-16T05:39:14.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.840+0000 7fb681400700 1 -- 172.21.15.27:0/157627906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb65c001000 con 0x7fb67c1037b0 2024-04-16T05:39:14.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.842+0000 7fb672a00700 1 -- 172.21.15.27:0/157627906 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fb668010850 con 0x7fb67c1037b0 2024-04-16T05:39:14.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.843+0000 7fb672a00700 1 --2- 172.21.15.27:0/157627906 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb6600692b0 0x7fb66006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:14.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.843+0000 7fb672a00700 1 -- 172.21.15.27:0/157627906 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb668087720 con 0x7fb67c1037b0 2024-04-16T05:39:14.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.844+0000 7fb67a800700 1 --2- 172.21.15.27:0/157627906 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb6600692b0 0x7fb66006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:14.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.844+0000 7fb672a00700 1 --2- 172.21.15.27:0/157627906 >> [v2:172.21.15.122:6800/186542313,v1:172.21.15.122:6801/186542313] conn(0x7fb66006ef80 0x7fb660071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:14.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.844+0000 7fb672a00700 1 -- 172.21.15.27:0/157627906 --> [v2:172.21.15.122:6800/186542313,v1:172.21.15.122:6801/186542313] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb660071a40 con 0x7fb66006ef80 2024-04-16T05:39:14.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.844+0000 7fb672a00700 1 -- 172.21.15.27:0/157627906 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb668087a60 con 0x7fb67c1037b0 2024-04-16T05:39:14.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.845+0000 7fb67b200700 1 --2- 172.21.15.27:0/157627906 >> [v2:172.21.15.122:6800/186542313,v1:172.21.15.122:6801/186542313] conn(0x7fb66006ef80 0x7fb660071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:14.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.845+0000 7fb67a800700 1 --2- 172.21.15.27:0/157627906 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb6600692b0 0x7fb66006b760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009fd0 tx=0x7fb66c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:14.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.846+0000 7fb67b200700 1 --2- 172.21.15.27:0/157627906 >> [v2:172.21.15.122:6800/186542313,v1:172.21.15.122:6801/186542313] conn(0x7fb66006ef80 0x7fb660071390 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:14.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.847+0000 7fb672a00700 1 -- 172.21.15.27:0/157627906 <== osd.3 v2:172.21.15.122:6800/186542313 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fb660071a40 con 0x7fb66006ef80 2024-04-16T05:39:14.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.866+0000 7fb681400700 1 -- 172.21.15.27:0/157627906 --> [v2:172.21.15.122:6800/186542313,v1:172.21.15.122:6801/186542313] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb65c002d60 con 0x7fb66006ef80 2024-04-16T05:39:14.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.867+0000 7fb672a00700 1 -- 172.21.15.27:0/157627906 <== osd.3 v2:172.21.15.122:6800/186542313 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb65c002d60 con 0x7fb66006ef80 2024-04-16T05:39:14.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.867+0000 7fb681400700 1 -- 172.21.15.27:0/157627906 >> [v2:172.21.15.122:6800/186542313,v1:172.21.15.122:6801/186542313] conn(0x7fb66006ef80 msgr2=0x7fb660071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:14.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.867+0000 7fb681400700 1 --2- 172.21.15.27:0/157627906 >> [v2:172.21.15.122:6800/186542313,v1:172.21.15.122:6801/186542313] conn(0x7fb66006ef80 0x7fb660071390 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.867+0000 7fb681400700 1 -- 172.21.15.27:0/157627906 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb6600692b0 msgr2=0x7fb66006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.867+0000 7fb681400700 1 --2- 172.21.15.27:0/157627906 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb6600692b0 0x7fb66006b760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009fd0 tx=0x7fb66c009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.867+0000 7fb681400700 1 -- 172.21.15.27:0/157627906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb67c1037b0 msgr2=0x7fb67c1162d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:14.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.867+0000 7fb681400700 1 --2- 172.21.15.27:0/157627906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb67c1037b0 0x7fb67c1162d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb66800c800 tx=0x7fb668004470 comp rx=0 tx=0).stop 2024-04-16T05:39:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.868+0000 7fb681400700 1 -- 172.21.15.27:0/157627906 shutdown_connections 2024-04-16T05:39:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.868+0000 7fb681400700 1 --2- 172.21.15.27:0/157627906 >> [v2:172.21.15.122:6800/186542313,v1:172.21.15.122:6801/186542313] conn(0x7fb66006ef80 0x7fb660071390 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.868+0000 7fb681400700 1 --2- 172.21.15.27:0/157627906 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb6600692b0 0x7fb66006b760 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.868+0000 7fb681400700 1 --2- 172.21.15.27:0/157627906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb67c102e10 0x7fb67c115d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.868+0000 7fb681400700 1 --2- 172.21.15.27:0/157627906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb67c1037b0 0x7fb67c1162d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.868+0000 7fb681400700 1 -- 172.21.15.27:0/157627906 >> 172.21.15.27:0/157627906 conn(0x7fb67c0f89a0 msgr2=0x7fb67c0fa670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.868+0000 7fb681400700 1 -- 172.21.15.27:0/157627906 shutdown_connections 2024-04-16T05:39:14.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:14.869+0000 7fb681400700 1 -- 172.21.15.27:0/157627906 wait complete. 2024-04-16T05:39:15.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:14 smithi122 ceph-mon[35998]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:15.575 INFO:teuthology.orchestra.run.smithi027.stdout:146028888081 2024-04-16T05:39:15.575 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-16T05:39:15.620 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:39:16.138 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:39:16.177 INFO:teuthology.orchestra.run.smithi027.stdout:111669149720 2024-04-16T05:39:16.178 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-16T05:39:16.229 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:39:16.710 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:16 smithi027 ceph-mon[29206]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:17.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:16 smithi122 ceph-mon[35998]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:18.109 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:39:18.561 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:39:18.562 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:39:18.897 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:18 smithi027 ceph-mon[29206]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:19.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:18 smithi122 ceph-mon[35998]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:19.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.391+0000 7fd21ba00700 1 -- 172.21.15.27:0/1666390074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd214102e20 msgr2=0x7fd2141031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:19.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.391+0000 7fd21ba00700 1 --2- 172.21.15.27:0/1666390074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd214102e20 0x7fd2141031f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd208004940 tx=0x7fd208009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:19.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.392+0000 7fd21ba00700 1 -- 172.21.15.27:0/1666390074 shutdown_connections 2024-04-16T05:39:19.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.392+0000 7fd21ba00700 1 --2- 172.21.15.27:0/1666390074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd2141037c0 0x7fd2140fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:19.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.392+0000 7fd21ba00700 1 --2- 172.21.15.27:0/1666390074 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd214102e20 0x7fd2141031f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:19.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.392+0000 7fd21ba00700 1 -- 172.21.15.27:0/1666390074 >> 172.21.15.27:0/1666390074 conn(0x7fd2140f89d0 msgr2=0x7fd2140fade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:19.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.392+0000 7fd21ba00700 1 -- 172.21.15.27:0/1666390074 shutdown_connections 2024-04-16T05:39:19.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.392+0000 7fd21ba00700 1 -- 172.21.15.27:0/1666390074 wait complete. 2024-04-16T05:39:19.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.393+0000 7fd21ba00700 1 Processor -- start 2024-04-16T05:39:19.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.393+0000 7fd21ba00700 1 -- start start 2024-04-16T05:39:19.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.394+0000 7fd21ba00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd214102e20 0x7fd214113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:19.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.394+0000 7fd21ba00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2141037c0 0x7fd2141140f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:19.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.394+0000 7fd21ba00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd214114740 con 0x7fd2141037c0 2024-04-16T05:39:19.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.394+0000 7fd21ba00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd21410db30 con 0x7fd214102e20 2024-04-16T05:39:19.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.394+0000 7fd213e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2141037c0 0x7fd2141140f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:19.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.394+0000 7fd213e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2141037c0 0x7fd2141140f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49712/0 (socket says 172.21.15.27:49712) 2024-04-16T05:39:19.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.394+0000 7fd213e00700 1 -- 172.21.15.27:0/3222426558 learned_addr learned my addr 172.21.15.27:0/3222426558 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:39:19.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.395+0000 7fd219000700 1 --2- 172.21.15.27:0/3222426558 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd214102e20 0x7fd214113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:19.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.395+0000 7fd213e00700 1 -- 172.21.15.27:0/3222426558 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd214102e20 msgr2=0x7fd214113bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:19.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.395+0000 7fd213e00700 1 --2- 172.21.15.27:0/3222426558 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd214102e20 0x7fd214113bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:19.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.395+0000 7fd213e00700 1 -- 172.21.15.27:0/3222426558 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd208005040 con 0x7fd2141037c0 2024-04-16T05:39:19.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.395+0000 7fd213e00700 1 --2- 172.21.15.27:0/3222426558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2141037c0 0x7fd2141140f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd200002a50 tx=0x7fd200004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:19.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.396+0000 7fd210c00700 1 -- 172.21.15.27:0/3222426558 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd200007cc0 con 0x7fd2141037c0 2024-04-16T05:39:19.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.396+0000 7fd210c00700 1 -- 172.21.15.27:0/3222426558 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd20000f040 con 0x7fd2141037c0 2024-04-16T05:39:19.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.396+0000 7fd21ba00700 1 -- 172.21.15.27:0/3222426558 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd21410de10 con 0x7fd2141037c0 2024-04-16T05:39:19.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.396+0000 7fd210c00700 1 -- 172.21.15.27:0/3222426558 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd20000d9c0 con 0x7fd2141037c0 2024-04-16T05:39:19.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.396+0000 7fd21ba00700 1 -- 172.21.15.27:0/3222426558 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd21410e3d0 con 0x7fd2141037c0 2024-04-16T05:39:19.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.397+0000 7fd21ba00700 1 -- 172.21.15.27:0/3222426558 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd214100130 con 0x7fd2141037c0 2024-04-16T05:39:19.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.398+0000 7fd210c00700 1 -- 172.21.15.27:0/3222426558 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fd200009730 con 0x7fd2141037c0 2024-04-16T05:39:19.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.400+0000 7fd210c00700 1 --2- 172.21.15.27:0/3222426558 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd1fc0692b0 0x7fd1fc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:19.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.400+0000 7fd210c00700 1 -- 172.21.15.27:0/3222426558 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd200086b50 con 0x7fd2141037c0 2024-04-16T05:39:19.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.401+0000 7fd219000700 1 --2- 172.21.15.27:0/3222426558 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd1fc0692b0 0x7fd1fc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:19.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.401+0000 7fd219000700 1 --2- 172.21.15.27:0/3222426558 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd1fc0692b0 0x7fd1fc06b760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd208009fd0 tx=0x7fd208009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:19.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.404+0000 7fd210c00700 1 -- 172.21.15.27:0/3222426558 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fd200056bb0 con 0x7fd2141037c0 2024-04-16T05:39:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.549+0000 7fdc16a00700 1 -- 172.21.15.27:0/236512079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc100656b0 msgr2=0x7fdc10065b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.549+0000 7fdc16a00700 1 --2- 172.21.15.27:0/236512079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc100656b0 0x7fdc10065b20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc004940 tx=0x7fdbfc009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.549+0000 7fdc16a00700 1 -- 172.21.15.27:0/236512079 shutdown_connections 2024-04-16T05:39:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.549+0000 7fdc16a00700 1 --2- 172.21.15.27:0/236512079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc100656b0 0x7fdc10065b20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.549+0000 7fdc16a00700 1 --2- 172.21.15.27:0/236512079 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc10102e60 0x7fdc10103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.549+0000 7fdc16a00700 1 -- 172.21.15.27:0/236512079 >> 172.21.15.27:0/236512079 conn(0x7fdc10072bc0 msgr2=0x7fdc10072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.549+0000 7fdc16a00700 1 -- 172.21.15.27:0/236512079 shutdown_connections 2024-04-16T05:39:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.549+0000 7fdc16a00700 1 -- 172.21.15.27:0/236512079 wait complete. 2024-04-16T05:39:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.550+0000 7fdc16a00700 1 Processor -- start 2024-04-16T05:39:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.550+0000 7fdc16a00700 1 -- start start 2024-04-16T05:39:19.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.550+0000 7fdc16a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc100656b0 0x7fdc10115d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.550+0000 7fdc16a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc10102e60 0x7fdc101162a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.550+0000 7fdc16a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc101168f0 con 0x7fdc10102e60 2024-04-16T05:39:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.551+0000 7fdc16a00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1010fcc0 con 0x7fdc100656b0 2024-04-16T05:39:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.551+0000 7fdc0f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc10102e60 0x7fdc101162a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.551+0000 7fdc0f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc10102e60 0x7fdc101162a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49726/0 (socket says 172.21.15.27:49726) 2024-04-16T05:39:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.551+0000 7fdc0f400700 1 -- 172.21.15.27:0/310529375 learned_addr learned my addr 172.21.15.27:0/310529375 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:39:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.551+0000 7fdc0fe00700 1 --2- 172.21.15.27:0/310529375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc100656b0 0x7fdc10115d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.551+0000 7fdc0f400700 1 -- 172.21.15.27:0/310529375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc100656b0 msgr2=0x7fdc10115d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.551+0000 7fdc0f400700 1 --2- 172.21.15.27:0/310529375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc100656b0 0x7fdc10115d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.551+0000 7fdc0f400700 1 -- 172.21.15.27:0/310529375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbfc005040 con 0x7fdc10102e60 2024-04-16T05:39:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.552+0000 7fdc0f400700 1 --2- 172.21.15.27:0/310529375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc10102e60 0x7fdc101162a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009150 tx=0x7fdbfc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.552+0000 7fdc07e00700 1 -- 172.21.15.27:0/310529375 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbfc00e040 con 0x7fdc10102e60 2024-04-16T05:39:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.552+0000 7fdc07e00700 1 -- 172.21.15.27:0/310529375 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdbfc013420 con 0x7fdc10102e60 2024-04-16T05:39:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.552+0000 7fdc16a00700 1 -- 172.21.15.27:0/310529375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc1010ff40 con 0x7fdc10102e60 2024-04-16T05:39:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.552+0000 7fdc07e00700 1 -- 172.21.15.27:0/310529375 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbfc01c630 con 0x7fdc10102e60 2024-04-16T05:39:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.552+0000 7fdc16a00700 1 -- 172.21.15.27:0/310529375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc10110500 con 0x7fdc10102e60 2024-04-16T05:39:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.554+0000 7fdc16a00700 1 -- 172.21.15.27:0/310529375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc10107320 con 0x7fdc10102e60 2024-04-16T05:39:19.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.555+0000 7fdc07e00700 1 -- 172.21.15.27:0/310529375 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fdbfc013590 con 0x7fdc10102e60 2024-04-16T05:39:19.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.556+0000 7fdc07e00700 1 --2- 172.21.15.27:0/310529375 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fdbf40692b0 0x7fdbf406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:19.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.556+0000 7fdc07e00700 1 -- 172.21.15.27:0/310529375 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdbfc088b90 con 0x7fdc10102e60 2024-04-16T05:39:19.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.556+0000 7fdc0fe00700 1 --2- 172.21.15.27:0/310529375 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fdbf40692b0 0x7fdbf406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:19.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.557+0000 7fdc0fe00700 1 --2- 172.21.15.27:0/310529375 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fdbf40692b0 0x7fdbf406b760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdc00004990 tx=0x7fdc00009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:19.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.560+0000 7fdc07e00700 1 -- 172.21.15.27:0/310529375 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fdbfc058d30 con 0x7fdc10102e60 2024-04-16T05:39:19.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.672+0000 7fd21ba00700 1 -- 172.21.15.27:0/3222426558 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fd2140fe570 con 0x7fd2141037c0 2024-04-16T05:39:19.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.673+0000 7fd210c00700 1 -- 172.21.15.27:0/3222426558 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd200056740 con 0x7fd2141037c0 2024-04-16T05:39:19.672 INFO:teuthology.orchestra.run.smithi027.stdout:38654705711 2024-04-16T05:39:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.675+0000 7fd21ba00700 1 -- 172.21.15.27:0/3222426558 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd1fc0692b0 msgr2=0x7fd1fc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.675+0000 7fd21ba00700 1 --2- 172.21.15.27:0/3222426558 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd1fc0692b0 0x7fd1fc06b760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd208009fd0 tx=0x7fd208009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.675+0000 7fd21ba00700 1 -- 172.21.15.27:0/3222426558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2141037c0 msgr2=0x7fd2141140f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:19.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.675+0000 7fd21ba00700 1 --2- 172.21.15.27:0/3222426558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2141037c0 0x7fd2141140f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd200002a50 tx=0x7fd200004600 comp rx=0 tx=0).stop 2024-04-16T05:39:19.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.676+0000 7fd21ba00700 1 -- 172.21.15.27:0/3222426558 shutdown_connections 2024-04-16T05:39:19.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.676+0000 7fd21ba00700 1 --2- 172.21.15.27:0/3222426558 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd1fc0692b0 0x7fd1fc06b760 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:19.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.676+0000 7fd21ba00700 1 --2- 172.21.15.27:0/3222426558 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd214102e20 0x7fd214113bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:19.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.676+0000 7fd21ba00700 1 --2- 172.21.15.27:0/3222426558 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd2141037c0 0x7fd2141140f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:19.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.676+0000 7fd21ba00700 1 -- 172.21.15.27:0/3222426558 >> 172.21.15.27:0/3222426558 conn(0x7fd2140f89d0 msgr2=0x7fd214107370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:19.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.676+0000 7fd21ba00700 1 -- 172.21.15.27:0/3222426558 shutdown_connections 2024-04-16T05:39:19.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.676+0000 7fd21ba00700 1 -- 172.21.15.27:0/3222426558 wait complete. 2024-04-16T05:39:19.698 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:19 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:39:19.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.793+0000 7fdc16a00700 1 -- 172.21.15.27:0/310529375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fdc100fe570 con 0x7fdc10102e60 2024-04-16T05:39:19.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.794+0000 7fdc07e00700 1 -- 172.21.15.27:0/310529375 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdbfc0588c0 con 0x7fdc10102e60 2024-04-16T05:39:19.795 INFO:teuthology.orchestra.run.smithi027.stdout:60129542184 2024-04-16T05:39:19.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.797+0000 7fdc16a00700 1 -- 172.21.15.27:0/310529375 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fdbf40692b0 msgr2=0x7fdbf406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:19.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.797+0000 7fdc16a00700 1 --2- 172.21.15.27:0/310529375 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fdbf40692b0 0x7fdbf406b760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdc00004990 tx=0x7fdc00009ee0 comp rx=0 tx=0).stop 2024-04-16T05:39:19.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.797+0000 7fdc16a00700 1 -- 172.21.15.27:0/310529375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc10102e60 msgr2=0x7fdc101162a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:19.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.797+0000 7fdc16a00700 1 --2- 172.21.15.27:0/310529375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc10102e60 0x7fdc101162a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009150 tx=0x7fdbfc009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:19.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.797+0000 7fdc16a00700 1 -- 172.21.15.27:0/310529375 shutdown_connections 2024-04-16T05:39:19.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.797+0000 7fdc16a00700 1 --2- 172.21.15.27:0/310529375 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fdbf40692b0 0x7fdbf406b760 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:19.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.797+0000 7fdc16a00700 1 --2- 172.21.15.27:0/310529375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdc100656b0 0x7fdc10115d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:19.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.797+0000 7fdc16a00700 1 --2- 172.21.15.27:0/310529375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdc10102e60 0x7fdc101162a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:19.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.797+0000 7fdc16a00700 1 -- 172.21.15.27:0/310529375 >> 172.21.15.27:0/310529375 conn(0x7fdc10072bc0 msgr2=0x7fdc100fad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:19.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.798+0000 7fdc16a00700 1 -- 172.21.15.27:0/310529375 shutdown_connections 2024-04-16T05:39:19.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:19.798+0000 7fdc16a00700 1 -- 172.21.15.27:0/310529375 wait complete. 2024-04-16T05:39:20.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:19 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:39:20.721 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:20 smithi027 ceph-mon[29206]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:20.722 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:20 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/3222426558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-16T05:39:20.722 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:20 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/310529375' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-16T05:39:21.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:20 smithi122 ceph-mon[35998]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:21.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:20 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/3222426558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-16T05:39:21.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:20 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/310529375' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-16T05:39:22.913 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:22 smithi027 ceph-mon[29206]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:23.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:22 smithi122 ceph-mon[35998]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:24.874 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:24 smithi027 ceph-mon[29206]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:25.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:24 smithi122 ceph-mon[35998]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:25.400 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542182 got 60129542184 for osd.1 2024-04-16T05:39:25.401 DEBUG:teuthology.parallel:result is None 2024-04-16T05:39:25.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.176+0000 7f6fec800700 1 -- 172.21.15.27:0/2510911768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe4071f90 msgr2=0x7f6fe4070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:25.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.176+0000 7f6fec800700 1 --2- 172.21.15.27:0/2510911768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe4071f90 0x7f6fe4070360 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8004970 tx=0x7f6fd8009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:25.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.176+0000 7f6fec800700 1 -- 172.21.15.27:0/2510911768 shutdown_connections 2024-04-16T05:39:25.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.176+0000 7f6fec800700 1 --2- 172.21.15.27:0/2510911768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fe40708a0 0x7f6fe4070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:25.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.176+0000 7f6fec800700 1 --2- 172.21.15.27:0/2510911768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe4071f90 0x7f6fe4070360 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:25.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.176+0000 7f6fec800700 1 -- 172.21.15.27:0/2510911768 >> 172.21.15.27:0/2510911768 conn(0x7f6fe40f89b0 msgr2=0x7f6fe40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:25.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.176+0000 7f6fec800700 1 -- 172.21.15.27:0/2510911768 shutdown_connections 2024-04-16T05:39:25.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.176+0000 7f6fec800700 1 -- 172.21.15.27:0/2510911768 wait complete. 2024-04-16T05:39:25.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.177+0000 7f6fec800700 1 Processor -- start 2024-04-16T05:39:25.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.178+0000 7f6fec800700 1 -- start start 2024-04-16T05:39:25.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.178+0000 7f6fec800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe40708a0 0x7f6fe411a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:25.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.178+0000 7f6fec800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fe4071f90 0x7f6fe411a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:25.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.178+0000 7f6fec800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe411aca0 con 0x7f6fe40708a0 2024-04-16T05:39:25.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.178+0000 7f6fec800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe4114120 con 0x7f6fe4071f90 2024-04-16T05:39:25.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.179+0000 7f6fe9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe40708a0 0x7f6fe411a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:25.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.179+0000 7f6fe9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe40708a0 0x7f6fe411a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41756/0 (socket says 172.21.15.27:41756) 2024-04-16T05:39:25.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.179+0000 7f6fe9e00700 1 -- 172.21.15.27:0/2424209871 learned_addr learned my addr 172.21.15.27:0/2424209871 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:39:25.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.179+0000 7f6fe9400700 1 --2- 172.21.15.27:0/2424209871 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fe4071f90 0x7f6fe411a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:25.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.179+0000 7f6fe9e00700 1 -- 172.21.15.27:0/2424209871 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fe4071f90 msgr2=0x7f6fe411a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:25.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.179+0000 7f6fe9e00700 1 --2- 172.21.15.27:0/2424209871 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fe4071f90 0x7f6fe411a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:25.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.179+0000 7f6fe9e00700 1 -- 172.21.15.27:0/2424209871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd8005040 con 0x7f6fe40708a0 2024-04-16T05:39:25.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.179+0000 7f6fe9e00700 1 --2- 172.21.15.27:0/2424209871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe40708a0 0x7f6fe411a110 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8000c00 tx=0x7f6fd8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:25.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.180+0000 7f6fde000700 1 -- 172.21.15.27:0/2424209871 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fd800e040 con 0x7f6fe40708a0 2024-04-16T05:39:25.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.180+0000 7f6fde000700 1 -- 172.21.15.27:0/2424209871 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6fd800bc10 con 0x7f6fe40708a0 2024-04-16T05:39:25.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.180+0000 7f6fec800700 1 -- 172.21.15.27:0/2424209871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe41143a0 con 0x7f6fe40708a0 2024-04-16T05:39:25.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.180+0000 7f6fde000700 1 -- 172.21.15.27:0/2424209871 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fd801c630 con 0x7f6fe40708a0 2024-04-16T05:39:25.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.180+0000 7f6fec800700 1 -- 172.21.15.27:0/2424209871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe41148e0 con 0x7f6fe40708a0 2024-04-16T05:39:25.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.181+0000 7f6fec800700 1 -- 172.21.15.27:0/2424209871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe4071aa0 con 0x7f6fe40708a0 2024-04-16T05:39:25.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.182+0000 7f6fde000700 1 -- 172.21.15.27:0/2424209871 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f6fd800bd80 con 0x7f6fe40708a0 2024-04-16T05:39:25.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.183+0000 7f6fde000700 1 --2- 172.21.15.27:0/2424209871 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f6fcc0692a0 0x7f6fcc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:25.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.183+0000 7f6fde000700 1 -- 172.21.15.27:0/2424209871 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6fd8088250 con 0x7f6fe40708a0 2024-04-16T05:39:25.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.183+0000 7f6fe9400700 1 --2- 172.21.15.27:0/2424209871 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f6fcc0692a0 0x7f6fcc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:25.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.184+0000 7f6fe9400700 1 --2- 172.21.15.27:0/2424209871 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f6fcc0692a0 0x7f6fcc06b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4007fd0 tx=0x7f6fd4009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:25.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.188+0000 7f6fde000700 1 -- 172.21.15.27:0/2424209871 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f6fd80548a0 con 0x7f6fe40708a0 2024-04-16T05:39:25.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.436+0000 7f6fec800700 1 -- 172.21.15.27:0/2424209871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6fe4110540 con 0x7f6fe40708a0 2024-04-16T05:39:25.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.437+0000 7f6fde000700 1 -- 172.21.15.27:0/2424209871 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6fd8054020 con 0x7f6fe40708a0 2024-04-16T05:39:25.437 INFO:teuthology.orchestra.run.smithi027.stdout:167503724556 2024-04-16T05:39:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.440+0000 7f6fec800700 1 -- 172.21.15.27:0/2424209871 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f6fcc0692a0 msgr2=0x7f6fcc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.440+0000 7f6fec800700 1 --2- 172.21.15.27:0/2424209871 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f6fcc0692a0 0x7f6fcc06b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4007fd0 tx=0x7f6fd4009040 comp rx=0 tx=0).stop 2024-04-16T05:39:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.440+0000 7f6fec800700 1 -- 172.21.15.27:0/2424209871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe40708a0 msgr2=0x7f6fe411a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.440+0000 7f6fec800700 1 --2- 172.21.15.27:0/2424209871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe40708a0 0x7f6fe411a110 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8000c00 tx=0x7f6fd8009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.440+0000 7f6fec800700 1 -- 172.21.15.27:0/2424209871 shutdown_connections 2024-04-16T05:39:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.440+0000 7f6fec800700 1 --2- 172.21.15.27:0/2424209871 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f6fcc0692a0 0x7f6fcc06b750 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.440+0000 7f6fec800700 1 --2- 172.21.15.27:0/2424209871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe40708a0 0x7f6fe411a110 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.440+0000 7f6fec800700 1 --2- 172.21.15.27:0/2424209871 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fe4071f90 0x7f6fe411a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.440+0000 7f6fec800700 1 -- 172.21.15.27:0/2424209871 >> 172.21.15.27:0/2424209871 conn(0x7f6fe40f89b0 msgr2=0x7f6fe40f9af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:25.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.441+0000 7f6fec800700 1 -- 172.21.15.27:0/2424209871 shutdown_connections 2024-04-16T05:39:25.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.441+0000 7f6fec800700 1 -- 172.21.15.27:0/2424209871 wait complete. 2024-04-16T05:39:25.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:25 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/332863394' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-16T05:39:25.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:25 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/2424209871' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-16T05:39:26.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:25 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/332863394' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-16T05:39:26.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:25 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/2424209871' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-16T05:39:26.313 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705709 got 38654705711 for osd.0 2024-04-16T05:39:26.314 DEBUG:teuthology.parallel:result is None 2024-04-16T05:39:26.639 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724554 got 167503724556 for osd.5 2024-04-16T05:39:26.639 DEBUG:teuthology.parallel:result is None 2024-04-16T05:39:26.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:26 smithi027 ceph-mon[29206]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:27.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:26 smithi122 ceph-mon[35998]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.761+0000 7f97f0a00700 1 -- 172.21.15.27:0/3251487680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97ec102cc0 msgr2=0x7f97ec103090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.761+0000 7f97f0a00700 1 --2- 172.21.15.27:0/3251487680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97ec102cc0 0x7f97ec103090 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f97dc004940 tx=0x7f97dc009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.761+0000 7f97f0a00700 1 -- 172.21.15.27:0/3251487680 shutdown_connections 2024-04-16T05:39:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.761+0000 7f97f0a00700 1 --2- 172.21.15.27:0/3251487680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97ec065670 0x7f97ec065ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.761+0000 7f97f0a00700 1 --2- 172.21.15.27:0/3251487680 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97ec102cc0 0x7f97ec103090 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.761+0000 7f97f0a00700 1 -- 172.21.15.27:0/3251487680 >> 172.21.15.27:0/3251487680 conn(0x7f97ec071b70 msgr2=0x7f97ec071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.761+0000 7f97f0a00700 1 -- 172.21.15.27:0/3251487680 shutdown_connections 2024-04-16T05:39:27.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.761+0000 7f97f0a00700 1 -- 172.21.15.27:0/3251487680 wait complete. 2024-04-16T05:39:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.762+0000 7f97f0a00700 1 Processor -- start 2024-04-16T05:39:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.762+0000 7f97f0a00700 1 -- start start 2024-04-16T05:39:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.763+0000 7f97f0a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97ec065670 0x7f97ec115c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.763+0000 7f97f0a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97ec102cc0 0x7f97ec116190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.763+0000 7f97f0a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec1167e0 con 0x7f97ec065670 2024-04-16T05:39:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.763+0000 7f97f0a00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec10fc00 con 0x7f97ec102cc0 2024-04-16T05:39:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.763+0000 7f97eb400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97ec065670 0x7f97ec115c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.763+0000 7f97eb400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97ec065670 0x7f97ec115c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41736/0 (socket says 172.21.15.27:41736) 2024-04-16T05:39:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.763+0000 7f97eb400700 1 -- 172.21.15.27:0/332863394 learned_addr learned my addr 172.21.15.27:0/332863394 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:39:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.763+0000 7f97eaa00700 1 --2- 172.21.15.27:0/332863394 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97ec102cc0 0x7f97ec116190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:27.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.763+0000 7f97eb400700 1 -- 172.21.15.27:0/332863394 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97ec102cc0 msgr2=0x7f97ec116190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:27.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.763+0000 7f97eb400700 1 --2- 172.21.15.27:0/332863394 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97ec102cc0 0x7f97ec116190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:27.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.763+0000 7f97eb400700 1 -- 172.21.15.27:0/332863394 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc005040 con 0x7f97ec065670 2024-04-16T05:39:27.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.764+0000 7f97eb400700 1 --2- 172.21.15.27:0/332863394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97ec065670 0x7f97ec115c50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009180 tx=0x7f97dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:27.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.764+0000 7f97e3400700 1 -- 172.21.15.27:0/332863394 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97dc00e040 con 0x7f97ec065670 2024-04-16T05:39:27.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.764+0000 7f97e3400700 1 -- 172.21.15.27:0/332863394 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f97dc00bbc0 con 0x7f97ec065670 2024-04-16T05:39:27.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.764+0000 7f97e3400700 1 -- 172.21.15.27:0/332863394 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97dc01c630 con 0x7f97ec065670 2024-04-16T05:39:27.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.764+0000 7f97f0a00700 1 -- 172.21.15.27:0/332863394 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ec10fe80 con 0x7f97ec065670 2024-04-16T05:39:27.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.764+0000 7f97f0a00700 1 -- 172.21.15.27:0/332863394 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ec110440 con 0x7f97ec065670 2024-04-16T05:39:27.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.765+0000 7f97f0a00700 1 -- 172.21.15.27:0/332863394 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97ec101690 con 0x7f97ec065670 2024-04-16T05:39:27.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.766+0000 7f97e3400700 1 -- 172.21.15.27:0/332863394 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f97dc00bd30 con 0x7f97ec065670 2024-04-16T05:39:27.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.767+0000 7f97e3400700 1 --2- 172.21.15.27:0/332863394 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f97d00692b0 0x7f97d006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:27.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.767+0000 7f97e3400700 1 -- 172.21.15.27:0/332863394 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f97dc088210 con 0x7f97ec065670 2024-04-16T05:39:27.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.767+0000 7f97eaa00700 1 --2- 172.21.15.27:0/332863394 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f97d00692b0 0x7f97d006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:27.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.768+0000 7f97eaa00700 1 --2- 172.21.15.27:0/332863394 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f97d00692b0 0x7f97d006b760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f97ec1114f0 tx=0x7f97d800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:27.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:24.773+0000 7f97e3400700 1 -- 172.21.15.27:0/332863394 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f97dc0582f0 con 0x7f97ec065670 2024-04-16T05:39:27.027 INFO:teuthology.orchestra.run.smithi027.stdout:111669149721 2024-04-16T05:39:27.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.002+0000 7f97f0a00700 1 -- 172.21.15.27:0/332863394 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f97ec0fe3b0 con 0x7f97ec065670 2024-04-16T05:39:27.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.003+0000 7f97e3400700 1 -- 172.21.15.27:0/332863394 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f97dc057e80 con 0x7f97ec065670 2024-04-16T05:39:27.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.005+0000 7f97f0a00700 1 -- 172.21.15.27:0/332863394 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f97d00692b0 msgr2=0x7f97d006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:27.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.005+0000 7f97f0a00700 1 --2- 172.21.15.27:0/332863394 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f97d00692b0 0x7f97d006b760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f97ec1114f0 tx=0x7f97d800b040 comp rx=0 tx=0).stop 2024-04-16T05:39:27.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.005+0000 7f97f0a00700 1 -- 172.21.15.27:0/332863394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97ec065670 msgr2=0x7f97ec115c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:27.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.005+0000 7f97f0a00700 1 --2- 172.21.15.27:0/332863394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97ec065670 0x7f97ec115c50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009180 tx=0x7f97dc009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:27.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.005+0000 7f97f0a00700 1 -- 172.21.15.27:0/332863394 shutdown_connections 2024-04-16T05:39:27.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.005+0000 7f97f0a00700 1 --2- 172.21.15.27:0/332863394 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f97d00692b0 0x7f97d006b760 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:27.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.006+0000 7f97f0a00700 1 --2- 172.21.15.27:0/332863394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97ec065670 0x7f97ec115c50 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:27.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.006+0000 7f97f0a00700 1 --2- 172.21.15.27:0/332863394 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f97ec102cc0 0x7f97ec116190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:27.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.006+0000 7f97f0a00700 1 -- 172.21.15.27:0/332863394 >> 172.21.15.27:0/332863394 conn(0x7f97ec071b70 msgr2=0x7f97ec0fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:27.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.006+0000 7f97f0a00700 1 -- 172.21.15.27:0/332863394 shutdown_connections 2024-04-16T05:39:27.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:25.006+0000 7f97f0a00700 1 -- 172.21.15.27:0/332863394 wait complete. 2024-04-16T05:39:27.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.474+0000 7f4e71600700 1 -- 172.21.15.27:0/3796160573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e6c070360 msgr2=0x7f4e6c070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:27.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.474+0000 7f4e71600700 1 --2- 172.21.15.27:0/3796160573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e6c070360 0x7f4e6c070730 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c004970 tx=0x7f4e5c009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:27.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.475+0000 7f4e71600700 1 -- 172.21.15.27:0/3796160573 shutdown_connections 2024-04-16T05:39:27.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.475+0000 7f4e71600700 1 --2- 172.21.15.27:0/3796160573 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e6c070c70 0x7f4e6c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:27.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.475+0000 7f4e71600700 1 --2- 172.21.15.27:0/3796160573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e6c070360 0x7f4e6c070730 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:27.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.475+0000 7f4e71600700 1 -- 172.21.15.27:0/3796160573 >> 172.21.15.27:0/3796160573 conn(0x7f4e6c0f89b0 msgr2=0x7f4e6c0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:27.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.475+0000 7f4e71600700 1 -- 172.21.15.27:0/3796160573 shutdown_connections 2024-04-16T05:39:27.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.475+0000 7f4e71600700 1 -- 172.21.15.27:0/3796160573 wait complete. 2024-04-16T05:39:27.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.476+0000 7f4e71600700 1 Processor -- start 2024-04-16T05:39:27.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.476+0000 7f4e71600700 1 -- start start 2024-04-16T05:39:27.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.476+0000 7f4e71600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e6c070360 0x7f4e6c115bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:27.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.476+0000 7f4e71600700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e6c070c70 0x7f4e6c116110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:27.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.476+0000 7f4e71600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c1167f0 con 0x7f4e6c070360 2024-04-16T05:39:27.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.476+0000 7f4e71600700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6c10fd40 con 0x7f4e6c070c70 2024-04-16T05:39:27.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.476+0000 7f4e6aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e6c070360 0x7f4e6c115bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:27.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.477+0000 7f4e6aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e6c070360 0x7f4e6c115bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41766/0 (socket says 172.21.15.27:41766) 2024-04-16T05:39:27.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.477+0000 7f4e6aa00700 1 -- 172.21.15.27:0/2544065628 learned_addr learned my addr 172.21.15.27:0/2544065628 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:39:27.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.477+0000 7f4e6a000700 1 --2- 172.21.15.27:0/2544065628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e6c070c70 0x7f4e6c116110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:27.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.477+0000 7f4e6aa00700 1 -- 172.21.15.27:0/2544065628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e6c070c70 msgr2=0x7f4e6c116110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:27.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.477+0000 7f4e6aa00700 1 --2- 172.21.15.27:0/2544065628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e6c070c70 0x7f4e6c116110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:27.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.477+0000 7f4e6aa00700 1 -- 172.21.15.27:0/2544065628 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c005040 con 0x7f4e6c070360 2024-04-16T05:39:27.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.477+0000 7f4e6a000700 1 --2- 172.21.15.27:0/2544065628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e6c070c70 0x7f4e6c116110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:39:27.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.477+0000 7f4e6aa00700 1 --2- 172.21.15.27:0/2544065628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e6c070360 0x7f4e6c115bd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c004f50 tx=0x7f4e5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:27.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.478+0000 7f4e62a00700 1 -- 172.21.15.27:0/2544065628 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e5c00e040 con 0x7f4e6c070360 2024-04-16T05:39:27.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.478+0000 7f4e62a00700 1 -- 172.21.15.27:0/2544065628 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e5c00bc10 con 0x7f4e6c070360 2024-04-16T05:39:27.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.478+0000 7f4e71600700 1 -- 172.21.15.27:0/2544065628 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e6c10ff60 con 0x7f4e6c070360 2024-04-16T05:39:27.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.478+0000 7f4e62a00700 1 -- 172.21.15.27:0/2544065628 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e5c01c630 con 0x7f4e6c070360 2024-04-16T05:39:27.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.478+0000 7f4e71600700 1 -- 172.21.15.27:0/2544065628 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e6c1104a0 con 0x7f4e6c070360 2024-04-16T05:39:27.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.479+0000 7f4e71600700 1 -- 172.21.15.27:0/2544065628 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e6c105f60 con 0x7f4e6c070360 2024-04-16T05:39:27.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.480+0000 7f4e62a00700 1 -- 172.21.15.27:0/2544065628 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f4e5c013420 con 0x7f4e6c070360 2024-04-16T05:39:27.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.481+0000 7f4e62a00700 1 --2- 172.21.15.27:0/2544065628 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f4e500692a0 0x7f4e5006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:27.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.481+0000 7f4e62a00700 1 -- 172.21.15.27:0/2544065628 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4e5c0880e0 con 0x7f4e6c070360 2024-04-16T05:39:27.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.481+0000 7f4e6a000700 1 --2- 172.21.15.27:0/2544065628 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f4e500692a0 0x7f4e5006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:27.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.482+0000 7f4e6a000700 1 --2- 172.21.15.27:0/2544065628 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f4e500692a0 0x7f4e5006b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4e58006fd0 tx=0x7f4e58008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:27.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.486+0000 7f4e62a00700 1 -- 172.21.15.27:0/2544065628 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f4e5c0581c0 con 0x7f4e6c070360 2024-04-16T05:39:27.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.708+0000 7f4e71600700 1 -- 172.21.15.27:0/2544065628 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f4e6c10c190 con 0x7f4e6c070360 2024-04-16T05:39:27.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.709+0000 7f4e62a00700 1 -- 172.21.15.27:0/2544065628 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4e5c057d50 con 0x7f4e6c070360 2024-04-16T05:39:27.707 INFO:teuthology.orchestra.run.smithi027.stdout:81604378658 2024-04-16T05:39:27.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.712+0000 7f4e71600700 1 -- 172.21.15.27:0/2544065628 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f4e500692a0 msgr2=0x7f4e5006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:27.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.712+0000 7f4e71600700 1 --2- 172.21.15.27:0/2544065628 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f4e500692a0 0x7f4e5006b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4e58006fd0 tx=0x7f4e58008040 comp rx=0 tx=0).stop 2024-04-16T05:39:27.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.712+0000 7f4e71600700 1 -- 172.21.15.27:0/2544065628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e6c070360 msgr2=0x7f4e6c115bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:27.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.712+0000 7f4e71600700 1 --2- 172.21.15.27:0/2544065628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e6c070360 0x7f4e6c115bd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c004f50 tx=0x7f4e5c009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:27.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.712+0000 7f4e71600700 1 -- 172.21.15.27:0/2544065628 shutdown_connections 2024-04-16T05:39:27.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.712+0000 7f4e71600700 1 --2- 172.21.15.27:0/2544065628 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f4e500692a0 0x7f4e5006b750 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:27.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.712+0000 7f4e71600700 1 --2- 172.21.15.27:0/2544065628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4e6c070360 0x7f4e6c115bd0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:27.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.712+0000 7f4e71600700 1 --2- 172.21.15.27:0/2544065628 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4e6c070c70 0x7f4e6c116110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:27.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.712+0000 7f4e71600700 1 -- 172.21.15.27:0/2544065628 >> 172.21.15.27:0/2544065628 conn(0x7f4e6c0f89b0 msgr2=0x7f4e6c1030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:27.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.712+0000 7f4e71600700 1 -- 172.21.15.27:0/2544065628 shutdown_connections 2024-04-16T05:39:27.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:27.712+0000 7f4e71600700 1 -- 172.21.15.27:0/2544065628 wait complete. 2024-04-16T05:39:28.594 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149720 got 111669149721 for osd.3 2024-04-16T05:39:28.594 DEBUG:teuthology.parallel:result is None 2024-04-16T05:39:29.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:28 smithi122 ceph-mon[35998]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:29.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:28 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/2544065628' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-16T05:39:29.034 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:28 smithi027 ceph-mon[29206]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:29.034 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:28 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/2544065628' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-16T05:39:29.337 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378655 got 81604378658 for osd.2 2024-04-16T05:39:29.337 DEBUG:teuthology.parallel:result is None 2024-04-16T05:39:29.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.905+0000 7fc0e9200700 1 -- 172.21.15.27:0/35271768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e4102cc0 msgr2=0x7fc0e4103090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:29.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.905+0000 7fc0e9200700 1 --2- 172.21.15.27:0/35271768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e4102cc0 0x7fc0e4103090 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4004940 tx=0x7fc0d4009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:29.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.905+0000 7fc0e9200700 1 -- 172.21.15.27:0/35271768 shutdown_connections 2024-04-16T05:39:29.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.905+0000 7fc0e9200700 1 --2- 172.21.15.27:0/35271768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0e4065670 0x7fc0e4065ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:29.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.905+0000 7fc0e9200700 1 --2- 172.21.15.27:0/35271768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e4102cc0 0x7fc0e4103090 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:29.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.905+0000 7fc0e9200700 1 -- 172.21.15.27:0/35271768 >> 172.21.15.27:0/35271768 conn(0x7fc0e4071b70 msgr2=0x7fc0e4071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:29.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.906+0000 7fc0e9200700 1 -- 172.21.15.27:0/35271768 shutdown_connections 2024-04-16T05:39:29.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.906+0000 7fc0e9200700 1 -- 172.21.15.27:0/35271768 wait complete. 2024-04-16T05:39:29.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.906+0000 7fc0e9200700 1 Processor -- start 2024-04-16T05:39:29.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.907+0000 7fc0e9200700 1 -- start start 2024-04-16T05:39:29.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.907+0000 7fc0e9200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e4065670 0x7fc0e4115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:29.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.907+0000 7fc0e9200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0e4102cc0 0x7fc0e41161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:29.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.907+0000 7fc0e9200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e41167f0 con 0x7fc0e4065670 2024-04-16T05:39:29.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.907+0000 7fc0e9200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e410fbc0 con 0x7fc0e4102cc0 2024-04-16T05:39:29.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.907+0000 7fc0e3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e4065670 0x7fc0e4115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:29.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.907+0000 7fc0e3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e4065670 0x7fc0e4115c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41784/0 (socket says 172.21.15.27:41784) 2024-04-16T05:39:29.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.907+0000 7fc0e3400700 1 -- 172.21.15.27:0/4001268234 learned_addr learned my addr 172.21.15.27:0/4001268234 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:39:29.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.908+0000 7fc0dbe00700 1 --2- 172.21.15.27:0/4001268234 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0e4102cc0 0x7fc0e41161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:29.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.908+0000 7fc0e3400700 1 -- 172.21.15.27:0/4001268234 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0e4102cc0 msgr2=0x7fc0e41161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:29.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.908+0000 7fc0e3400700 1 --2- 172.21.15.27:0/4001268234 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0e4102cc0 0x7fc0e41161a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:29.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.908+0000 7fc0e3400700 1 -- 172.21.15.27:0/4001268234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d4005040 con 0x7fc0e4065670 2024-04-16T05:39:29.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.908+0000 7fc0e3400700 1 --2- 172.21.15.27:0/4001268234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e4065670 0x7fc0e4115c60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009180 tx=0x7fc0d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:29.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.908+0000 7fc0db400700 1 -- 172.21.15.27:0/4001268234 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0d400e040 con 0x7fc0e4065670 2024-04-16T05:39:29.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.909+0000 7fc0db400700 1 -- 172.21.15.27:0/4001268234 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc0d400bbc0 con 0x7fc0e4065670 2024-04-16T05:39:29.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.909+0000 7fc0e9200700 1 -- 172.21.15.27:0/4001268234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e410fe40 con 0x7fc0e4065670 2024-04-16T05:39:29.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.909+0000 7fc0db400700 1 -- 172.21.15.27:0/4001268234 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0d401c630 con 0x7fc0e4065670 2024-04-16T05:39:29.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.909+0000 7fc0e9200700 1 -- 172.21.15.27:0/4001268234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e4110400 con 0x7fc0e4065670 2024-04-16T05:39:29.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.910+0000 7fc0e9200700 1 -- 172.21.15.27:0/4001268234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0e4101690 con 0x7fc0e4065670 2024-04-16T05:39:29.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.911+0000 7fc0db400700 1 -- 172.21.15.27:0/4001268234 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fc0d400bd30 con 0x7fc0e4065670 2024-04-16T05:39:29.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.912+0000 7fc0db400700 1 --2- 172.21.15.27:0/4001268234 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc0c80692b0 0x7fc0c806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:29.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.912+0000 7fc0db400700 1 -- 172.21.15.27:0/4001268234 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc0d4088320 con 0x7fc0e4065670 2024-04-16T05:39:29.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.912+0000 7fc0dbe00700 1 --2- 172.21.15.27:0/4001268234 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc0c80692b0 0x7fc0c806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:29.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.913+0000 7fc0dbe00700 1 --2- 172.21.15.27:0/4001268234 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc0c80692b0 0x7fc0c806b760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc007fd0 tx=0x7fc0cc009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:29.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:29.917+0000 7fc0db400700 1 -- 172.21.15.27:0/4001268234 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fc0d4058380 con 0x7fc0e4065670 2024-04-16T05:39:30.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:30.134+0000 7fc0e9200700 1 -- 172.21.15.27:0/4001268234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fc0e40fe3b0 con 0x7fc0e4065670 2024-04-16T05:39:30.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:30.135+0000 7fc0db400700 1 -- 172.21.15.27:0/4001268234 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc0d4057f10 con 0x7fc0e4065670 2024-04-16T05:39:30.134 INFO:teuthology.orchestra.run.smithi027.stdout:146028888085 2024-04-16T05:39:30.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:30.138+0000 7fc0e9200700 1 -- 172.21.15.27:0/4001268234 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc0c80692b0 msgr2=0x7fc0c806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:30.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:30.138+0000 7fc0e9200700 1 --2- 172.21.15.27:0/4001268234 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc0c80692b0 0x7fc0c806b760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc007fd0 tx=0x7fc0cc009040 comp rx=0 tx=0).stop 2024-04-16T05:39:30.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:30.138+0000 7fc0e9200700 1 -- 172.21.15.27:0/4001268234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e4065670 msgr2=0x7fc0e4115c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:30.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:30.138+0000 7fc0e9200700 1 --2- 172.21.15.27:0/4001268234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e4065670 0x7fc0e4115c60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009180 tx=0x7fc0d4009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:30.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:30.139+0000 7fc0e9200700 1 -- 172.21.15.27:0/4001268234 shutdown_connections 2024-04-16T05:39:30.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:30.139+0000 7fc0e9200700 1 --2- 172.21.15.27:0/4001268234 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc0c80692b0 0x7fc0c806b760 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:30.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:30.139+0000 7fc0e9200700 1 --2- 172.21.15.27:0/4001268234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0e4065670 0x7fc0e4115c60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:30.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:30.139+0000 7fc0e9200700 1 --2- 172.21.15.27:0/4001268234 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0e4102cc0 0x7fc0e41161a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:30.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:30.139+0000 7fc0e9200700 1 -- 172.21.15.27:0/4001268234 >> 172.21.15.27:0/4001268234 conn(0x7fc0e4071b70 msgr2=0x7fc0e40fe810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:30.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:30.139+0000 7fc0e9200700 1 -- 172.21.15.27:0/4001268234 shutdown_connections 2024-04-16T05:39:30.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:30.139+0000 7fc0e9200700 1 -- 172.21.15.27:0/4001268234 wait complete. 2024-04-16T05:39:30.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:30 smithi027 ceph-mon[29206]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:30.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:30 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/4001268234' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-16T05:39:31.004 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888081 got 146028888085 for osd.4 2024-04-16T05:39:31.004 DEBUG:teuthology.parallel:result is None 2024-04-16T05:39:31.004 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-16T05:39:31.004 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-16T05:39:31.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:30 smithi122 ceph-mon[35998]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:31.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:30 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/4001268234' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-16T05:39:31.529 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:39:32.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:32 smithi027 ceph-mon[29206]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:33.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:32 smithi122 ceph-mon[35998]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:33.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.853+0000 7f2a65000700 1 -- 172.21.15.27:0/1455168494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a60109280 msgr2=0x7f2a60109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:33.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.853+0000 7f2a65000700 1 --2- 172.21.15.27:0/1455168494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a60109280 0x7f2a60109650 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2a50004940 tx=0x7f2a50009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:33.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.853+0000 7f2a65000700 1 -- 172.21.15.27:0/1455168494 shutdown_connections 2024-04-16T05:39:33.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.853+0000 7f2a65000700 1 --2- 172.21.15.27:0/1455168494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a60109c20 0x7f2a60103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:33.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.853+0000 7f2a65000700 1 --2- 172.21.15.27:0/1455168494 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a60109280 0x7f2a60109650 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:33.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.853+0000 7f2a65000700 1 -- 172.21.15.27:0/1455168494 >> 172.21.15.27:0/1455168494 conn(0x7f2a60072a60 msgr2=0x7f2a60074e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:33.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.854+0000 7f2a65000700 1 -- 172.21.15.27:0/1455168494 shutdown_connections 2024-04-16T05:39:33.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.854+0000 7f2a65000700 1 -- 172.21.15.27:0/1455168494 wait complete. 2024-04-16T05:39:33.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.854+0000 7f2a65000700 1 Processor -- start 2024-04-16T05:39:33.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.855+0000 7f2a65000700 1 -- start start 2024-04-16T05:39:33.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.855+0000 7f2a65000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a60109280 0x7f2a6011a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.855+0000 7f2a65000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a60109c20 0x7f2a6011a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.855+0000 7f2a65000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6011aba0 con 0x7f2a60109c20 2024-04-16T05:39:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.855+0000 7f2a65000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a60113f70 con 0x7f2a60109280 2024-04-16T05:39:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.855+0000 7f2a5ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a60109c20 0x7f2a6011a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.856+0000 7f2a5ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a60109c20 0x7f2a6011a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41794/0 (socket says 172.21.15.27:41794) 2024-04-16T05:39:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.856+0000 7f2a5ea00700 1 -- 172.21.15.27:0/3118301459 learned_addr learned my addr 172.21.15.27:0/3118301459 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:39:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.856+0000 7f2a5f400700 1 --2- 172.21.15.27:0/3118301459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a60109280 0x7f2a6011a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.856+0000 7f2a5ea00700 1 -- 172.21.15.27:0/3118301459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a60109280 msgr2=0x7f2a6011a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.856+0000 7f2a5ea00700 1 --2- 172.21.15.27:0/3118301459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a60109280 0x7f2a6011a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.856+0000 7f2a5ea00700 1 -- 172.21.15.27:0/3118301459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a50005040 con 0x7f2a60109c20 2024-04-16T05:39:33.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.857+0000 7f2a5ea00700 1 --2- 172.21.15.27:0/3118301459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a60109c20 0x7f2a6011a550 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c002a50 tx=0x7f2a4c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.857+0000 7f2a57400700 1 -- 172.21.15.27:0/3118301459 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a4c007cc0 con 0x7f2a60109c20 2024-04-16T05:39:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.857+0000 7f2a65000700 1 -- 172.21.15.27:0/3118301459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a60114250 con 0x7f2a60109c20 2024-04-16T05:39:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.857+0000 7f2a57400700 1 -- 172.21.15.27:0/3118301459 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a4c00f040 con 0x7f2a60109c20 2024-04-16T05:39:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.857+0000 7f2a57400700 1 -- 172.21.15.27:0/3118301459 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a4c00d9c0 con 0x7f2a60109c20 2024-04-16T05:39:33.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.857+0000 7f2a65000700 1 -- 172.21.15.27:0/3118301459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a60114810 con 0x7f2a60109c20 2024-04-16T05:39:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.858+0000 7f2a65000700 1 -- 172.21.15.27:0/3118301459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a601049d0 con 0x7f2a60109c20 2024-04-16T05:39:33.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.859+0000 7f2a57400700 1 -- 172.21.15.27:0/3118301459 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f2a4c00db40 con 0x7f2a60109c20 2024-04-16T05:39:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.860+0000 7f2a57400700 1 --2- 172.21.15.27:0/3118301459 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f2a440692a0 0x7f2a4406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:33.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.860+0000 7f2a57400700 1 -- 172.21.15.27:0/3118301459 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2a4c086cb0 con 0x7f2a60109c20 2024-04-16T05:39:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.860+0000 7f2a5f400700 1 --2- 172.21.15.27:0/3118301459 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f2a440692a0 0x7f2a4406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:33.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.861+0000 7f2a5f400700 1 --2- 172.21.15.27:0/3118301459 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f2a440692a0 0x7f2a4406b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2a50004f90 tx=0x7f2a50009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:33.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:33.865+0000 7f2a57400700 1 -- 172.21.15.27:0/3118301459 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f2a4c056c90 con 0x7f2a60109c20 2024-04-16T05:39:34.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:34.076+0000 7f2a65000700 1 -- 172.21.15.27:0/3118301459 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2a601103e0 con 0x7f2a440692a0 2024-04-16T05:39:34.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:34.078+0000 7f2a57400700 1 -- 172.21.15.27:0/3118301459 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27343 (secure 0 0 0) 0x7f2a601103e0 con 0x7f2a440692a0 2024-04-16T05:39:34.077 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:39:34.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:34.081+0000 7f2a65000700 1 -- 172.21.15.27:0/3118301459 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f2a440692a0 msgr2=0x7f2a4406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:34.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:34.081+0000 7f2a65000700 1 --2- 172.21.15.27:0/3118301459 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f2a440692a0 0x7f2a4406b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2a50004f90 tx=0x7f2a50009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:34.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:34.081+0000 7f2a65000700 1 -- 172.21.15.27:0/3118301459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a60109c20 msgr2=0x7f2a6011a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:34.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:34.081+0000 7f2a65000700 1 --2- 172.21.15.27:0/3118301459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a60109c20 0x7f2a6011a550 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c002a50 tx=0x7f2a4c004600 comp rx=0 tx=0).stop 2024-04-16T05:39:34.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:34.081+0000 7f2a65000700 1 -- 172.21.15.27:0/3118301459 shutdown_connections 2024-04-16T05:39:34.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:34.081+0000 7f2a65000700 1 --2- 172.21.15.27:0/3118301459 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f2a440692a0 0x7f2a4406b750 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:34.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:34.081+0000 7f2a65000700 1 --2- 172.21.15.27:0/3118301459 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a60109280 0x7f2a6011a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:34.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:34.081+0000 7f2a65000700 1 --2- 172.21.15.27:0/3118301459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a60109c20 0x7f2a6011a550 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:34.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:34.081+0000 7f2a65000700 1 -- 172.21.15.27:0/3118301459 >> 172.21.15.27:0/3118301459 conn(0x7f2a60072a60 msgr2=0x7f2a60074a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:34.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:34.081+0000 7f2a65000700 1 -- 172.21.15.27:0/3118301459 shutdown_connections 2024-04-16T05:39:34.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:34.081+0000 7f2a65000700 1 -- 172.21.15.27:0/3118301459 wait complete. 2024-04-16T05:39:34.081 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-16T05:39:35.004 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:34 smithi027 ceph-mon[29206]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:35.004 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:34 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:39:35.007 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-04-16T05:39:33.570015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788024,"kb_used_data":3876,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682888,"statfs":{"total":575970213888,"available":574139277312,"internally_reserved":0,"allocated":3969024,"data_stored":2606262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006812"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-16T05:38:30.705380+0000","last_change":"2024-04-16T05:37:57.628722+0000","last_active":"2024-04-16T05:38:30.705380+0000","last_peered":"2024-04-16T05:38:30.705380+0000","last_clean":"2024-04-16T05:38:30.705380+0000","last_became_active":"2024-04-16T05:37:57.628131+0000","last_became_peered":"2024-04-16T05:37:57.628131+0000","last_unstale":"2024-04-16T05:38:30.705380+0000","last_undegraded":"2024-04-16T05:38:30.705380+0000","last_fullsized":"2024-04-16T05:38:30.705380+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T05:36:52.436231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T05:36:52.436231+0000","last_clean_scrub_stamp":"2024-04-16T05:36:52.436231+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T10:56:47.230295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.446}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]}]},{"osd":4,"up_from":34,"seq":146028888086,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297716,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 05:39:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.8640000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.2010000000000001}]},{"osd":1,"last update":"Tue Apr 16 05:39:00 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":1.151},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.9410000000000001,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":0.95399999999999996}]},{"osd":2,"last update":"Tue Apr 16 05:39:00 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":1.101}]},{"osd":3,"last update":"Tue Apr 16 05:39:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.256}]},{"osd":5,"last update":"Tue Apr 16 05:39:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.98099999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149723,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 05:39:26 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.97499999999999998,"15min":0.97099999999999997},"min":{"1min":0.71299999999999997,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.0269999999999999,"5min":0.95599999999999996,"15min":0.94399999999999995},"min":{"1min":0.69499999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.77400000000000002}]},{"osd":1,"last update":"Tue Apr 16 05:39:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.008,"15min":0.99399999999999999},"min":{"1min":0.82499999999999996,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.04,"15min":1.036},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.284,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.84099999999999997}]},{"osd":2,"last update":"Tue Apr 16 05:39:26 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.0149999999999999,"15min":1.0009999999999999},"min":{"1min":0.72299999999999998,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.3360000000000001,"5min":1.397,"15min":1.397},"last":0.999},{"interface":"front","average":{"1min":1.224,"5min":1.0620000000000001,"15min":1.0349999999999999},"min":{"1min":0.83799999999999997,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.92000000000000004}]},{"osd":4,"last update":"Tue Apr 16 05:38:58 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.706,"5min":1.706,"15min":1.706},"last":1.248}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1950000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378659,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 05:38:54 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.85699999999999998,"15min":0.85499999999999998},"min":{"1min":0.65100000000000002,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.097,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.016,"5min":0.93400000000000005,"15min":0.92000000000000004},"min":{"1min":0.64500000000000002,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.54,"5min":1.54,"15min":1.54},"last":1.1379999999999999}]},{"osd":1,"last update":"Tue Apr 16 05:38:54 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.2869999999999999,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.052,"5min":1.018,"15min":1.012},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.4019999999999999,"5min":1.722,"15min":1.722},"last":1.1830000000000001}]},{"osd":3,"last update":"Tue Apr 16 05:39:28 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.133,"15min":1.1399999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3520000000000001,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":1.482},{"interface":"front","average":{"1min":1.246,"5min":1.1479999999999999,"15min":1.1319999999999999},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":2.0470000000000002,"5min":2.0470000000000002,"15min":2.0470000000000002},"last":1.5489999999999999}]},{"osd":4,"last update":"Tue Apr 16 05:39:00 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":1.423}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.671}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 16 05:39:17 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.76400000000000001,"15min":0.71399999999999997},"min":{"1min":0.45900000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.621},{"interface":"front","average":{"1min":1.075,"5min":0.83899999999999997,"15min":0.755},"min":{"1min":0.55200000000000005,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":2.431,"5min":2.431,"15min":2.431},"last":1.319}]},{"osd":2,"last update":"Tue Apr 16 05:38:57 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.0549999999999999,"15min":1.0369999999999999},"min":{"1min":0.61399999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.0649999999999999,"5min":0.96799999999999997,"15min":0.95199999999999996},"min":{"1min":0.51600000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.52,"5min":1.637,"15min":1.637},"last":0.77000000000000002}]},{"osd":3,"last update":"Tue Apr 16 05:39:26 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.2709999999999999,"15min":1.2450000000000001},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":2.1619999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.673},{"interface":"front","average":{"1min":1.246,"5min":1.1859999999999999,"15min":1.1759999999999999},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.96399999999999997}]},{"osd":4,"last update":"Tue Apr 16 05:38:57 2024","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.371,"15min":1.371},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":4.3979999999999997,"5min":4.3979999999999997,"15min":4.3979999999999997},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.282,"5min":1.282,"15min":1.282},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":1.635,"5min":1.635,"15min":1.635},"last":1.2669999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.444}]}]},{"osd":1,"up_from":14,"seq":60129542187,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 05:39:14 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.80800000000000005,"15min":0.75700000000000001},"min":{"1min":0.52000000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.631,"5min":1.631,"15min":1.631},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.1100000000000001,"5min":0.879,"15min":0.80900000000000005},"min":{"1min":0.53300000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":1.333}]},{"osd":2,"last update":"Tue Apr 16 05:38:58 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.016,"15min":1.01},"min":{"1min":0.69799999999999995,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.048,"5min":1.01,"15min":1.004},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":1.2869999999999999}]},{"osd":3,"last update":"Tue Apr 16 05:39:28 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.226,"15min":1.2310000000000001},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.77,"5min":3.1760000000000002,"15min":3.1760000000000002},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.252,"5min":1.2529999999999999,"15min":1.2529999999999999},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":1.903,"5min":1.903,"15min":1.903},"last":1.2050000000000001}]},{"osd":4,"last update":"Tue Apr 16 05:38:58 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":1.026,"5min":1.026,"15min":1.026},"max":{"1min":3.1469999999999998,"5min":3.1469999999999998,"15min":3.1469999999999998},"last":1.415},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.109,"5min":2.109,"15min":2.109},"last":1.7110000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.514}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T05:39:35.008 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-16T05:39:35.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:34 smithi122 ceph-mon[35998]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:35.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:34 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:39:35.524 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:39:36.003 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:35 smithi027 ceph-mon[29206]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-16T05:39:36.003 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:35 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:39:36.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:35 smithi122 ceph-mon[35998]: from='client.14406 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-16T05:39:36.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:35 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:39:37.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:36 smithi122 ceph-mon[35998]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:37.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:36 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:39:37.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:36 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:39:37.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:36 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:39:37.023 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:36 smithi027 ceph-mon[29206]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:37.023 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:36 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:39:37.024 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:36 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:39:37.024 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:36 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:39:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.812+0000 7fd10f400700 1 -- 172.21.15.27:0/1313084771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1081059d0 msgr2=0x7fd1080ff090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.812+0000 7fd10f400700 1 --2- 172.21.15.27:0/1313084771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1081059d0 0x7fd1080ff090 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4004940 tx=0x7fd0f4009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.812+0000 7fd10f400700 1 -- 172.21.15.27:0/1313084771 shutdown_connections 2024-04-16T05:39:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.812+0000 7fd10f400700 1 --2- 172.21.15.27:0/1313084771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1081059d0 0x7fd1080ff090 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.812+0000 7fd10f400700 1 --2- 172.21.15.27:0/1313084771 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd108105030 0x7fd108105400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:37.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.812+0000 7fd10f400700 1 -- 172.21.15.27:0/1313084771 >> 172.21.15.27:0/1313084771 conn(0x7fd1080fabd0 msgr2=0x7fd1080fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:37.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.812+0000 7fd10f400700 1 -- 172.21.15.27:0/1313084771 shutdown_connections 2024-04-16T05:39:37.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.812+0000 7fd10f400700 1 -- 172.21.15.27:0/1313084771 wait complete. 2024-04-16T05:39:37.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.813+0000 7fd10f400700 1 Processor -- start 2024-04-16T05:39:37.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.813+0000 7fd10f400700 1 -- start start 2024-04-16T05:39:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.813+0000 7fd10f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd108105030 0x7fd108115b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.813+0000 7fd10f400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1081059d0 0x7fd1081160d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:37.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.813+0000 7fd10f400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd108116720 con 0x7fd108105030 2024-04-16T05:39:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.813+0000 7fd10f400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10810fd00 con 0x7fd1081059d0 2024-04-16T05:39:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.814+0000 7fd10ca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd108105030 0x7fd108115b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.814+0000 7fd10ca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd108105030 0x7fd108115b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33698/0 (socket says 172.21.15.27:33698) 2024-04-16T05:39:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.814+0000 7fd10ca00700 1 -- 172.21.15.27:0/1943028604 learned_addr learned my addr 172.21.15.27:0/1943028604 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:39:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.814+0000 7fd107e00700 1 --2- 172.21.15.27:0/1943028604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1081059d0 0x7fd1081160d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.814+0000 7fd10ca00700 1 -- 172.21.15.27:0/1943028604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1081059d0 msgr2=0x7fd1081160d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.814+0000 7fd10ca00700 1 --2- 172.21.15.27:0/1943028604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1081059d0 0x7fd1081160d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.814+0000 7fd10ca00700 1 -- 172.21.15.27:0/1943028604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f4005040 con 0x7fd108105030 2024-04-16T05:39:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.814+0000 7fd107e00700 1 --2- 172.21.15.27:0/1943028604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1081059d0 0x7fd1081160d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:39:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.815+0000 7fd10ca00700 1 --2- 172.21.15.27:0/1943028604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd108105030 0x7fd108115b90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc002a50 tx=0x7fd0fc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.815+0000 7fd104c00700 1 -- 172.21.15.27:0/1943028604 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0fc007cc0 con 0x7fd108105030 2024-04-16T05:39:37.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.815+0000 7fd104c00700 1 -- 172.21.15.27:0/1943028604 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd0fc00f040 con 0x7fd108105030 2024-04-16T05:39:37.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.815+0000 7fd10f400700 1 -- 172.21.15.27:0/1943028604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd10810ffe0 con 0x7fd108105030 2024-04-16T05:39:37.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.815+0000 7fd104c00700 1 -- 172.21.15.27:0/1943028604 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0fc009640 con 0x7fd108105030 2024-04-16T05:39:37.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.815+0000 7fd10f400700 1 -- 172.21.15.27:0/1943028604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1081105a0 con 0x7fd108105030 2024-04-16T05:39:37.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.816+0000 7fd10f400700 1 -- 172.21.15.27:0/1943028604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd108100780 con 0x7fd108105030 2024-04-16T05:39:37.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.817+0000 7fd104c00700 1 -- 172.21.15.27:0/1943028604 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fd0fc0097a0 con 0x7fd108105030 2024-04-16T05:39:37.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.818+0000 7fd104c00700 1 --2- 172.21.15.27:0/1943028604 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd0f00692a0 0x7fd0f006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:37.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.818+0000 7fd104c00700 1 -- 172.21.15.27:0/1943028604 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd0fc086c90 con 0x7fd108105030 2024-04-16T05:39:37.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.819+0000 7fd107e00700 1 --2- 172.21.15.27:0/1943028604 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd0f00692a0 0x7fd0f006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:37.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.819+0000 7fd107e00700 1 --2- 172.21.15.27:0/1943028604 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd0f00692a0 0x7fd0f006b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009fd0 tx=0x7fd0f4009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:37.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:37.823+0000 7fd104c00700 1 -- 172.21.15.27:0/1943028604 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fd0fc056d10 con 0x7fd108105030 2024-04-16T05:39:38.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:38.033+0000 7fd10f400700 1 -- 172.21.15.27:0/1943028604 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd108111250 con 0x7fd0f00692a0 2024-04-16T05:39:38.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:38.035+0000 7fd104c00700 1 -- 172.21.15.27:0/1943028604 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+31093 (secure 0 0 0) 0x7fd108111250 con 0x7fd0f00692a0 2024-04-16T05:39:38.034 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:39:38.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:38.038+0000 7fd10f400700 1 -- 172.21.15.27:0/1943028604 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd0f00692a0 msgr2=0x7fd0f006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:38.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:38.038+0000 7fd10f400700 1 --2- 172.21.15.27:0/1943028604 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd0f00692a0 0x7fd0f006b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009fd0 tx=0x7fd0f4009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:38.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:38.038+0000 7fd10f400700 1 -- 172.21.15.27:0/1943028604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd108105030 msgr2=0x7fd108115b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:38.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:38.038+0000 7fd10f400700 1 --2- 172.21.15.27:0/1943028604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd108105030 0x7fd108115b90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc002a50 tx=0x7fd0fc0046b0 comp rx=0 tx=0).stop 2024-04-16T05:39:38.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:38.039+0000 7fd10f400700 1 -- 172.21.15.27:0/1943028604 shutdown_connections 2024-04-16T05:39:38.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:38.039+0000 7fd10f400700 1 --2- 172.21.15.27:0/1943028604 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd0f00692a0 0x7fd0f006b750 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:38.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:38.039+0000 7fd10f400700 1 --2- 172.21.15.27:0/1943028604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd108105030 0x7fd108115b90 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:38.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:38.039+0000 7fd10f400700 1 --2- 172.21.15.27:0/1943028604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1081059d0 0x7fd1081160d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:38.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:38.039+0000 7fd10f400700 1 -- 172.21.15.27:0/1943028604 >> 172.21.15.27:0/1943028604 conn(0x7fd1080fabd0 msgr2=0x7fd1080fbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:38.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:38.039+0000 7fd10f400700 1 -- 172.21.15.27:0/1943028604 shutdown_connections 2024-04-16T05:39:38.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:38.039+0000 7fd10f400700 1 -- 172.21.15.27:0/1943028604 wait complete. 2024-04-16T05:39:38.039 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-04-16T05:39:38.971 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:38 smithi027 ceph-mon[29206]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:38.976 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-04-16T05:39:37.570997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788024,"kb_used_data":3876,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682888,"statfs":{"total":575970213888,"available":574139277312,"internally_reserved":0,"allocated":3969024,"data_stored":2606262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002891"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-16T05:38:30.705380+0000","last_change":"2024-04-16T05:37:57.628722+0000","last_active":"2024-04-16T05:38:30.705380+0000","last_peered":"2024-04-16T05:38:30.705380+0000","last_clean":"2024-04-16T05:38:30.705380+0000","last_became_active":"2024-04-16T05:37:57.628131+0000","last_became_peered":"2024-04-16T05:37:57.628131+0000","last_unstale":"2024-04-16T05:38:30.705380+0000","last_undegraded":"2024-04-16T05:38:30.705380+0000","last_fullsized":"2024-04-16T05:38:30.705380+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T05:36:52.436231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T05:36:52.436231+0000","last_clean_scrub_stamp":"2024-04-16T05:36:52.436231+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T10:56:47.230295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724559,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 05:39:34 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":1.194},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.0390000000000001},"last":0.89500000000000002}]},{"osd":1,"last update":"Tue Apr 16 05:39:34 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.109,"15min":1.109},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":1.044}]},{"osd":2,"last update":"Tue Apr 16 05:39:34 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.121,"15min":1.121},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":1.004},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":1.272}]},{"osd":3,"last update":"Tue Apr 16 05:39:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.117},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.0680000000000001}]},{"osd":4,"last update":"Tue Apr 16 05:39:34 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":1.167},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.238}]}]},{"osd":4,"up_from":34,"seq":146028888086,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297716,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 05:39:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.8640000000000001,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":1.2010000000000001}]},{"osd":1,"last update":"Tue Apr 16 05:39:00 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.024,"15min":1.024},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":1.151},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":1.9410000000000001,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":0.95399999999999996}]},{"osd":2,"last update":"Tue Apr 16 05:39:00 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":1.502,"5min":1.502,"15min":1.502},"last":1.101}]},{"osd":3,"last update":"Tue Apr 16 05:39:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.256}]},{"osd":5,"last update":"Tue Apr 16 05:39:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.98099999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149724,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 05:39:26 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.97499999999999998,"15min":0.97099999999999997},"min":{"1min":0.71299999999999997,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.0269999999999999,"5min":0.95599999999999996,"15min":0.94399999999999995},"min":{"1min":0.69499999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.78300000000000003}]},{"osd":1,"last update":"Tue Apr 16 05:39:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.008,"15min":0.99399999999999999},"min":{"1min":0.82499999999999996,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":1.004},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.04,"15min":1.036},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.284,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":1.036}]},{"osd":2,"last update":"Tue Apr 16 05:39:26 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.0149999999999999,"15min":1.0009999999999999},"min":{"1min":0.72299999999999998,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.3360000000000001,"5min":1.397,"15min":1.397},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.224,"5min":1.0620000000000001,"15min":1.0349999999999999},"min":{"1min":0.83799999999999997,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":1.2370000000000001}]},{"osd":4,"last update":"Tue Apr 16 05:38:58 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.706,"5min":1.706,"15min":1.706},"last":1.4239999999999999}]},{"osd":5,"last update":"Tue Apr 16 05:39:31 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":1.302,"15min":1.302},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.579,"5min":1.579,"15min":1.579},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":1.1819999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378660,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 05:38:54 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.85699999999999998,"15min":0.85499999999999998},"min":{"1min":0.65100000000000002,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.097,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.016,"5min":0.93400000000000005,"15min":0.92000000000000004},"min":{"1min":0.64500000000000002,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.54,"5min":1.54,"15min":1.54},"last":1.0009999999999999}]},{"osd":1,"last update":"Tue Apr 16 05:38:54 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97499999999999998,"15min":0.97499999999999998},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.2869999999999999,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.052,"5min":1.018,"15min":1.012},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.4019999999999999,"5min":1.722,"15min":1.722},"last":0.70899999999999996}]},{"osd":3,"last update":"Tue Apr 16 05:39:28 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.133,"15min":1.1399999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3520000000000001,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.246,"5min":1.1479999999999999,"15min":1.1319999999999999},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":2.0470000000000002,"5min":2.0470000000000002,"15min":2.0470000000000002},"last":1.036}]},{"osd":4,"last update":"Tue Apr 16 05:39:00 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":1.1870000000000001}]},{"osd":5,"last update":"Tue Apr 16 05:39:33 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.1720000000000002,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":1.1259999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705715,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 16 05:39:17 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.76400000000000001,"15min":0.71399999999999997},"min":{"1min":0.45900000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.075,"5min":0.83899999999999997,"15min":0.755},"min":{"1min":0.55200000000000005,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":2.431,"5min":2.431,"15min":2.431},"last":0.77700000000000002}]},{"osd":2,"last update":"Tue Apr 16 05:38:57 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.0549999999999999,"15min":1.0369999999999999},"min":{"1min":0.61399999999999999,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.0649999999999999,"5min":0.96799999999999997,"15min":0.95199999999999996},"min":{"1min":0.51600000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.52,"5min":1.637,"15min":1.637},"last":0.94199999999999995}]},{"osd":3,"last update":"Tue Apr 16 05:39:26 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.2709999999999999,"15min":1.2450000000000001},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":2.1619999999999999,"5min":2.1619999999999999,"15min":2.1619999999999999},"last":1.266},{"interface":"front","average":{"1min":1.246,"5min":1.1859999999999999,"15min":1.1759999999999999},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.137}]},{"osd":4,"last update":"Tue Apr 16 05:38:57 2024","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.371,"15min":1.371},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":4.3979999999999997,"5min":4.3979999999999997,"15min":4.3979999999999997},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.282,"5min":1.282,"15min":1.282},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"max":{"1min":1.635,"5min":1.635,"15min":1.635},"last":1.0269999999999999}]},{"osd":5,"last update":"Tue Apr 16 05:39:34 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":3.8730000000000002,"5min":3.8730000000000002,"15min":3.8730000000000002},"last":1.196},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.2829999999999999,"5min":2.2829999999999999,"15min":2.2829999999999999},"last":1.3640000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 16 05:39:14 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.80800000000000005,"15min":0.75700000000000001},"min":{"1min":0.52000000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.631,"5min":1.631,"15min":1.631},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.1100000000000001,"5min":0.879,"15min":0.80900000000000005},"min":{"1min":0.53300000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.98099999999999998}]},{"osd":2,"last update":"Tue Apr 16 05:38:58 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.016,"15min":1.01},"min":{"1min":0.69799999999999995,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.048,"5min":1.01,"15min":1.004},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":1.169}]},{"osd":3,"last update":"Tue Apr 16 05:39:28 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.226,"15min":1.2310000000000001},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.77,"5min":3.1760000000000002,"15min":3.1760000000000002},"last":1.222},{"interface":"front","average":{"1min":1.252,"5min":1.2529999999999999,"15min":1.2529999999999999},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"max":{"1min":1.903,"5min":1.903,"15min":1.903},"last":1.4610000000000001}]},{"osd":4,"last update":"Tue Apr 16 05:38:58 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":1.026,"5min":1.026,"15min":1.026},"max":{"1min":3.1469999999999998,"5min":3.1469999999999998,"15min":3.1469999999999998},"last":1.635},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":2.109,"5min":2.109,"15min":2.109},"last":1.089}]},{"osd":5,"last update":"Tue Apr 16 05:39:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.7270000000000001,"5min":1.7270000000000001,"15min":1.7270000000000001},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":1.3069999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-16T05:39:38.977 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-16T05:39:38.978 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-16T05:39:38.978 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-16T05:39:38.978 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph health --format=json 2024-04-16T05:39:39.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:38 smithi122 ceph-mon[35998]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:39.505 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:39:39.917 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:39 smithi027 ceph-mon[29206]: from='client.14410 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-16T05:39:40.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:39 smithi122 ceph-mon[35998]: from='client.14410 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-16T05:39:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:40 smithi027 ceph-mon[29206]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:41.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:40 smithi122 ceph-mon[35998]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:41.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.856+0000 7fc4dae00700 1 -- 172.21.15.27:0/1451985432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4d4072e00 msgr2=0x7fc4d410d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:41.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.856+0000 7fc4dae00700 1 --2- 172.21.15.27:0/1451985432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4d4072e00 0x7fc4d410d5e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0004940 tx=0x7fc4c0009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:41.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.856+0000 7fc4dae00700 1 -- 172.21.15.27:0/1451985432 shutdown_connections 2024-04-16T05:39:41.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.856+0000 7fc4dae00700 1 --2- 172.21.15.27:0/1451985432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4d4072e00 0x7fc4d410d5e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:41.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.856+0000 7fc4dae00700 1 --2- 172.21.15.27:0/1451985432 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4d40724f0 0x7fc4d40728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:41.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.856+0000 7fc4dae00700 1 -- 172.21.15.27:0/1451985432 >> 172.21.15.27:0/1451985432 conn(0x7fc4d40fab10 msgr2=0x7fc4d40fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:41.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.856+0000 7fc4dae00700 1 -- 172.21.15.27:0/1451985432 shutdown_connections 2024-04-16T05:39:41.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.856+0000 7fc4dae00700 1 -- 172.21.15.27:0/1451985432 wait complete. 2024-04-16T05:39:41.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.857+0000 7fc4dae00700 1 Processor -- start 2024-04-16T05:39:41.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.857+0000 7fc4dae00700 1 -- start start 2024-04-16T05:39:41.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.858+0000 7fc4dae00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4d40724f0 0x7fc4d4119fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:41.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.858+0000 7fc4dae00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4d4072e00 0x7fc4d411a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:41.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.858+0000 7fc4dae00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d411ab50 con 0x7fc4d4072e00 2024-04-16T05:39:41.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.858+0000 7fc4dae00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d4113f20 con 0x7fc4d40724f0 2024-04-16T05:39:41.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.858+0000 7fc4d9000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4d4072e00 0x7fc4d411a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:41.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.858+0000 7fc4d9000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4d4072e00 0x7fc4d411a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33722/0 (socket says 172.21.15.27:33722) 2024-04-16T05:39:41.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.858+0000 7fc4d9000700 1 -- 172.21.15.27:0/3463464396 learned_addr learned my addr 172.21.15.27:0/3463464396 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:39:41.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.858+0000 7fc4d9a00700 1 --2- 172.21.15.27:0/3463464396 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4d40724f0 0x7fc4d4119fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:41.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.858+0000 7fc4d9000700 1 -- 172.21.15.27:0/3463464396 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4d40724f0 msgr2=0x7fc4d4119fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:41.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.858+0000 7fc4d9000700 1 --2- 172.21.15.27:0/3463464396 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4d40724f0 0x7fc4d4119fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:41.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.859+0000 7fc4d9000700 1 -- 172.21.15.27:0/3463464396 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c0005040 con 0x7fc4d4072e00 2024-04-16T05:39:41.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.859+0000 7fc4d9000700 1 --2- 172.21.15.27:0/3463464396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4d4072e00 0x7fc4d411a500 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0004890 tx=0x7fc4c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:41.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.859+0000 7fc4d1600700 1 -- 172.21.15.27:0/3463464396 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4c000e040 con 0x7fc4d4072e00 2024-04-16T05:39:41.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.859+0000 7fc4d1600700 1 -- 172.21.15.27:0/3463464396 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc4c0009af0 con 0x7fc4d4072e00 2024-04-16T05:39:41.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.859+0000 7fc4d1600700 1 -- 172.21.15.27:0/3463464396 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4c001c630 con 0x7fc4d4072e00 2024-04-16T05:39:41.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.859+0000 7fc4dae00700 1 -- 172.21.15.27:0/3463464396 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d41141a0 con 0x7fc4d4072e00 2024-04-16T05:39:41.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.860+0000 7fc4dae00700 1 -- 172.21.15.27:0/3463464396 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d4114760 con 0x7fc4d4072e00 2024-04-16T05:39:41.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.861+0000 7fc4dae00700 1 -- 172.21.15.27:0/3463464396 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4d4073c30 con 0x7fc4d4072e00 2024-04-16T05:39:41.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.862+0000 7fc4d1600700 1 -- 172.21.15.27:0/3463464396 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fc4c000bbc0 con 0x7fc4d4072e00 2024-04-16T05:39:41.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.862+0000 7fc4d1600700 1 --2- 172.21.15.27:0/3463464396 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc4bc0692a0 0x7fc4bc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:41.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.863+0000 7fc4d1600700 1 -- 172.21.15.27:0/3463464396 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc4c0088120 con 0x7fc4d4072e00 2024-04-16T05:39:41.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.863+0000 7fc4d9a00700 1 --2- 172.21.15.27:0/3463464396 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc4bc0692a0 0x7fc4bc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:41.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.863+0000 7fc4d9a00700 1 --2- 172.21.15.27:0/3463464396 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc4bc0692a0 0x7fc4bc06b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8004550 tx=0x7fc4c8009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:41.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:41.867+0000 7fc4d1600700 1 -- 172.21.15.27:0/3463464396 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fc4c0058200 con 0x7fc4d4072e00 2024-04-16T05:39:42.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:41 smithi122 ceph-mon[35998]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:42.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:41 smithi027 ceph-mon[29206]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:42.132+0000 7fc4dae00700 1 -- 172.21.15.27:0/3463464396 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc4d405f5f0 con 0x7fc4d4072e00 2024-04-16T05:39:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:42.133+0000 7fc4d1600700 1 -- 172.21.15.27:0/3463464396 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fc4c0057d90 con 0x7fc4d4072e00 2024-04-16T05:39:42.132 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:39:42.132 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-16T05:39:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:42.136+0000 7fc4dae00700 1 -- 172.21.15.27:0/3463464396 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc4bc0692a0 msgr2=0x7fc4bc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:42.136+0000 7fc4dae00700 1 --2- 172.21.15.27:0/3463464396 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc4bc0692a0 0x7fc4bc06b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8004550 tx=0x7fc4c8009040 comp rx=0 tx=0).stop 2024-04-16T05:39:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:42.136+0000 7fc4dae00700 1 -- 172.21.15.27:0/3463464396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4d4072e00 msgr2=0x7fc4d411a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:42.136+0000 7fc4dae00700 1 --2- 172.21.15.27:0/3463464396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4d4072e00 0x7fc4d411a500 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0004890 tx=0x7fc4c0009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:42.136+0000 7fc4dae00700 1 -- 172.21.15.27:0/3463464396 shutdown_connections 2024-04-16T05:39:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:42.136+0000 7fc4dae00700 1 --2- 172.21.15.27:0/3463464396 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc4bc0692a0 0x7fc4bc06b750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:42.136+0000 7fc4dae00700 1 --2- 172.21.15.27:0/3463464396 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4d40724f0 0x7fc4d4119fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:42.136+0000 7fc4dae00700 1 --2- 172.21.15.27:0/3463464396 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4d4072e00 0x7fc4d411a500 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:42.136+0000 7fc4dae00700 1 -- 172.21.15.27:0/3463464396 >> 172.21.15.27:0/3463464396 conn(0x7fc4d40fab10 msgr2=0x7fc4d40fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:42.137+0000 7fc4dae00700 1 -- 172.21.15.27:0/3463464396 shutdown_connections 2024-04-16T05:39:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:42.137+0000 7fc4dae00700 1 -- 172.21.15.27:0/3463464396 wait complete. 2024-04-16T05:39:42.996 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:42 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/3463464396' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-16T05:39:43.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:42 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/3463464396' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-16T05:39:43.087 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-16T05:39:43.087 INFO:tasks.cephadm:Setup complete, yielding 2024-04-16T05:39:43.088 INFO:teuthology.run_tasks:Running task print... 2024-04-16T05:39:43.099 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-04-16T05:39:43.100 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T05:39:43.111 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-16T05:39:43.111 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-04-16T05:39:43.638 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:39:44.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:43 smithi122 ceph-mon[35998]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:44.025 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:43 smithi027 ceph-mon[29206]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.926+0000 7fec98600700 1 -- 172.21.15.27:0/236184373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec900fce80 msgr2=0x7fec900fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.926+0000 7fec98600700 1 --2- 172.21.15.27:0/236184373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec900fce80 0x7fec900fd290 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fec84004940 tx=0x7fec84009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.926+0000 7fec98600700 1 -- 172.21.15.27:0/236184373 shutdown_connections 2024-04-16T05:39:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.926+0000 7fec98600700 1 --2- 172.21.15.27:0/236184373 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec900fdff0 0x7fec900fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.926+0000 7fec98600700 1 --2- 172.21.15.27:0/236184373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec900fce80 0x7fec900fd290 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.926+0000 7fec98600700 1 -- 172.21.15.27:0/236184373 >> 172.21.15.27:0/236184373 conn(0x7fec900f8580 msgr2=0x7fec900fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.926+0000 7fec98600700 1 -- 172.21.15.27:0/236184373 shutdown_connections 2024-04-16T05:39:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.926+0000 7fec98600700 1 -- 172.21.15.27:0/236184373 wait complete. 2024-04-16T05:39:45.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.927+0000 7fec98600700 1 Processor -- start 2024-04-16T05:39:45.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.927+0000 7fec98600700 1 -- start start 2024-04-16T05:39:45.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.928+0000 7fec98600700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec900fce80 0x7fec9010fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:45.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.928+0000 7fec98600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec900fdff0 0x7fec90110090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:45.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.928+0000 7fec98600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec90110690 con 0x7fec900fdff0 2024-04-16T05:39:45.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.928+0000 7fec98600700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec90110800 con 0x7fec900fce80 2024-04-16T05:39:45.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.928+0000 7fec95200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec900fdff0 0x7fec90110090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:45.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.928+0000 7fec95200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec900fdff0 0x7fec90110090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46016/0 (socket says 172.21.15.27:46016) 2024-04-16T05:39:45.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.928+0000 7fec95200700 1 -- 172.21.15.27:0/3655110991 learned_addr learned my addr 172.21.15.27:0/3655110991 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:39:45.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.929+0000 7fec95c00700 1 --2- 172.21.15.27:0/3655110991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec900fce80 0x7fec9010fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:45.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.929+0000 7fec95200700 1 -- 172.21.15.27:0/3655110991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec900fce80 msgr2=0x7fec9010fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:45.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.929+0000 7fec95200700 1 --2- 172.21.15.27:0/3655110991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec900fce80 0x7fec9010fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:45.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.929+0000 7fec95200700 1 -- 172.21.15.27:0/3655110991 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec84005040 con 0x7fec900fdff0 2024-04-16T05:39:45.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.929+0000 7fec95c00700 1 --2- 172.21.15.27:0/3655110991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec900fce80 0x7fec9010fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T05:39:45.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.929+0000 7fec95200700 1 --2- 172.21.15.27:0/3655110991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec900fdff0 0x7fec90110090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fec7c002a50 tx=0x7fec7c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:45.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.930+0000 7fec8d600700 1 -- 172.21.15.27:0/3655110991 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec7c007cc0 con 0x7fec900fdff0 2024-04-16T05:39:45.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.930+0000 7fec98600700 1 -- 172.21.15.27:0/3655110991 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec90115260 con 0x7fec900fdff0 2024-04-16T05:39:45.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.930+0000 7fec8d600700 1 -- 172.21.15.27:0/3655110991 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec7c00f040 con 0x7fec900fdff0 2024-04-16T05:39:45.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.930+0000 7fec8d600700 1 -- 172.21.15.27:0/3655110991 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec7c009640 con 0x7fec900fdff0 2024-04-16T05:39:45.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.930+0000 7fec98600700 1 -- 172.21.15.27:0/3655110991 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec90115820 con 0x7fec900fdff0 2024-04-16T05:39:45.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.931+0000 7fec98600700 1 -- 172.21.15.27:0/3655110991 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec90101980 con 0x7fec900fdff0 2024-04-16T05:39:45.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.932+0000 7fec8d600700 1 -- 172.21.15.27:0/3655110991 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fec7c0097a0 con 0x7fec900fdff0 2024-04-16T05:39:45.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.933+0000 7fec8d600700 1 --2- 172.21.15.27:0/3655110991 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fec780692a0 0x7fec7806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:45.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.933+0000 7fec8d600700 1 -- 172.21.15.27:0/3655110991 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fec7c086c20 con 0x7fec900fdff0 2024-04-16T05:39:45.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.934+0000 7fec95c00700 1 --2- 172.21.15.27:0/3655110991 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fec780692a0 0x7fec7806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:45.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.934+0000 7fec95c00700 1 --2- 172.21.15.27:0/3655110991 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fec780692a0 0x7fec7806b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fec8400b3b0 tx=0x7fec84009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:45.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:45.938+0000 7fec8d600700 1 -- 172.21.15.27:0/3655110991 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fec7c056c80 con 0x7fec900fdff0 2024-04-16T05:39:46.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:46.157+0000 7fec98600700 1 -- 172.21.15.27:0/3655110991 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7fec9004c710 con 0x7fec900fdff0 2024-04-16T05:39:46.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:46.164+0000 7fec8d600700 1 -- 172.21.15.27:0/3655110991 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7fec7c056810 con 0x7fec900fdff0 2024-04-16T05:39:46.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:46.167+0000 7fec98600700 1 -- 172.21.15.27:0/3655110991 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fec780692a0 msgr2=0x7fec7806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:46.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:46.167+0000 7fec98600700 1 --2- 172.21.15.27:0/3655110991 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fec780692a0 0x7fec7806b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fec8400b3b0 tx=0x7fec84009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:46.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:46.167+0000 7fec98600700 1 -- 172.21.15.27:0/3655110991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec900fdff0 msgr2=0x7fec90110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:46.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:46.167+0000 7fec98600700 1 --2- 172.21.15.27:0/3655110991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec900fdff0 0x7fec90110090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fec7c002a50 tx=0x7fec7c0046b0 comp rx=0 tx=0).stop 2024-04-16T05:39:46.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:46.167+0000 7fec98600700 1 -- 172.21.15.27:0/3655110991 shutdown_connections 2024-04-16T05:39:46.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:46.167+0000 7fec98600700 1 --2- 172.21.15.27:0/3655110991 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fec780692a0 0x7fec7806b750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:46.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:46.167+0000 7fec98600700 1 --2- 172.21.15.27:0/3655110991 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fec900fce80 0x7fec9010fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:46.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:46.167+0000 7fec98600700 1 --2- 172.21.15.27:0/3655110991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec900fdff0 0x7fec90110090 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:46.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:46.167+0000 7fec98600700 1 -- 172.21.15.27:0/3655110991 >> 172.21.15.27:0/3655110991 conn(0x7fec900f8580 msgr2=0x7fec900f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:46.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:46.168+0000 7fec98600700 1 -- 172.21.15.27:0/3655110991 shutdown_connections 2024-04-16T05:39:46.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:46.168+0000 7fec98600700 1 -- 172.21.15.27:0/3655110991 wait complete. 2024-04-16T05:39:46.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:46 smithi027 ceph-mon[29206]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:46.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:46 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/3655110991' entity='client.admin' 2024-04-16T05:39:46.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:46 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:39:46.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:46 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:39:46.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:46 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:39:46.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:46 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:39:47.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:46 smithi122 ceph-mon[35998]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:47.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:46 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/3655110991' entity='client.admin' 2024-04-16T05:39:47.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:46 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:39:47.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:46 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:39:47.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:46 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:39:47.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:46 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:39:47.088 INFO:teuthology.run_tasks:Running task print... 2024-04-16T05:39:47.097 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-04-16T05:39:47.097 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T05:39:47.112 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-16T05:39:47.112 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch status' 2024-04-16T05:39:47.644 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:39:49.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:48 smithi122 ceph-mon[35998]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:49.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:48 smithi027 ceph-mon[29206]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:49.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.944+0000 7f869b800700 1 -- 172.21.15.27:0/3421619836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86940fcdc0 msgr2=0x7f86940fd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:49.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.944+0000 7f869b800700 1 --2- 172.21.15.27:0/3421619836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86940fcdc0 0x7f86940fd1d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8688004940 tx=0x7f8688009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:49.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.944+0000 7f869b800700 1 -- 172.21.15.27:0/3421619836 shutdown_connections 2024-04-16T05:39:49.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.944+0000 7f869b800700 1 --2- 172.21.15.27:0/3421619836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86940fd710 0x7f86940fe070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:49.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.944+0000 7f869b800700 1 --2- 172.21.15.27:0/3421619836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86940fcdc0 0x7f86940fd1d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:49.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.944+0000 7f869b800700 1 -- 172.21.15.27:0/3421619836 >> 172.21.15.27:0/3421619836 conn(0x7f86940f8540 msgr2=0x7f86940fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:49.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.944+0000 7f869b800700 1 -- 172.21.15.27:0/3421619836 shutdown_connections 2024-04-16T05:39:49.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.944+0000 7f869b800700 1 -- 172.21.15.27:0/3421619836 wait complete. 2024-04-16T05:39:49.945 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:49 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:39:49.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.945+0000 7f869b800700 1 Processor -- start 2024-04-16T05:39:49.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.946+0000 7f869b800700 1 -- start start 2024-04-16T05:39:49.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.946+0000 7f869b800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86940fcdc0 0x7f86941140b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:49.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.946+0000 7f869b800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86940fd710 0x7f86941145f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:49.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.946+0000 7f869b800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8694114bf0 con 0x7f86940fd710 2024-04-16T05:39:49.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.946+0000 7f869b800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86941153f0 con 0x7f86940fcdc0 2024-04-16T05:39:49.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.946+0000 7f8693e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86940fd710 0x7f86941145f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:49.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.946+0000 7f8693e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86940fd710 0x7f86941145f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46036/0 (socket says 172.21.15.27:46036) 2024-04-16T05:39:49.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.946+0000 7f8693e00700 1 -- 172.21.15.27:0/2262682251 learned_addr learned my addr 172.21.15.27:0/2262682251 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:39:49.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.947+0000 7f8698e00700 1 --2- 172.21.15.27:0/2262682251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86940fcdc0 0x7f86941140b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:49.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.947+0000 7f8693e00700 1 -- 172.21.15.27:0/2262682251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86940fcdc0 msgr2=0x7f86941140b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:49.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.947+0000 7f8693e00700 1 --2- 172.21.15.27:0/2262682251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86940fcdc0 0x7f86941140b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:49.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.947+0000 7f8693e00700 1 -- 172.21.15.27:0/2262682251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8688005040 con 0x7f86940fd710 2024-04-16T05:39:49.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.947+0000 7f8698e00700 1 --2- 172.21.15.27:0/2262682251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86940fcdc0 0x7f86941140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T05:39:49.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.947+0000 7f8693e00700 1 --2- 172.21.15.27:0/2262682251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86940fd710 0x7f86941145f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f868000c800 tx=0x7f8680004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:49.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.947+0000 7f8690c00700 1 -- 172.21.15.27:0/2262682251 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f868000a760 con 0x7f86940fd710 2024-04-16T05:39:49.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.947+0000 7f869b800700 1 -- 172.21.15.27:0/2262682251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8694115670 con 0x7f86940fd710 2024-04-16T05:39:49.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.947+0000 7f8690c00700 1 -- 172.21.15.27:0/2262682251 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f868000ad90 con 0x7f86940fd710 2024-04-16T05:39:49.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.948+0000 7f869b800700 1 -- 172.21.15.27:0/2262682251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8694115c30 con 0x7f86940fd710 2024-04-16T05:39:49.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.948+0000 7f8690c00700 1 -- 172.21.15.27:0/2262682251 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f868001b400 con 0x7f86940fd710 2024-04-16T05:39:49.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.949+0000 7f869b800700 1 -- 172.21.15.27:0/2262682251 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8694101a60 con 0x7f86940fd710 2024-04-16T05:39:49.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.950+0000 7f8690c00700 1 -- 172.21.15.27:0/2262682251 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f8680012400 con 0x7f86940fd710 2024-04-16T05:39:49.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.951+0000 7f8690c00700 1 --2- 172.21.15.27:0/2262682251 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f867c0692b0 0x7f867c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:49.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.951+0000 7f8690c00700 1 -- 172.21.15.27:0/2262682251 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8680086f60 con 0x7f86940fd710 2024-04-16T05:39:49.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.951+0000 7f8698e00700 1 --2- 172.21.15.27:0/2262682251 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f867c0692b0 0x7f867c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:49.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.952+0000 7f8698e00700 1 --2- 172.21.15.27:0/2262682251 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f867c0692b0 0x7f867c06b760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8688000c00 tx=0x7f8688009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:49.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:49.955+0000 7f8690c00700 1 -- 172.21.15.27:0/2262682251 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f8680056fc0 con 0x7f86940fd710 2024-04-16T05:39:50.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:49 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:39:50.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:50.188+0000 7f869b800700 1 -- 172.21.15.27:0/2262682251 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8694115f00 con 0x7f867c0692b0 2024-04-16T05:39:50.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:50.191+0000 7f8690c00700 1 -- 172.21.15.27:0/2262682251 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f8694115f00 con 0x7f867c0692b0 2024-04-16T05:39:50.190 INFO:teuthology.orchestra.run.smithi027.stdout:Backend: cephadm 2024-04-16T05:39:50.191 INFO:teuthology.orchestra.run.smithi027.stdout:Available: Yes 2024-04-16T05:39:50.191 INFO:teuthology.orchestra.run.smithi027.stdout:Paused: No 2024-04-16T05:39:50.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:50.194+0000 7f869b800700 1 -- 172.21.15.27:0/2262682251 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f867c0692b0 msgr2=0x7f867c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:50.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:50.194+0000 7f869b800700 1 --2- 172.21.15.27:0/2262682251 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f867c0692b0 0x7f867c06b760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8688000c00 tx=0x7f8688009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:50.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:50.194+0000 7f869b800700 1 -- 172.21.15.27:0/2262682251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86940fd710 msgr2=0x7f86941145f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:50.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:50.194+0000 7f869b800700 1 --2- 172.21.15.27:0/2262682251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86940fd710 0x7f86941145f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f868000c800 tx=0x7f8680004470 comp rx=0 tx=0).stop 2024-04-16T05:39:50.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:50.194+0000 7f869b800700 1 -- 172.21.15.27:0/2262682251 shutdown_connections 2024-04-16T05:39:50.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:50.194+0000 7f869b800700 1 --2- 172.21.15.27:0/2262682251 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f867c0692b0 0x7f867c06b760 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:50.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:50.194+0000 7f869b800700 1 --2- 172.21.15.27:0/2262682251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86940fcdc0 0x7f86941140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:50.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:50.194+0000 7f869b800700 1 --2- 172.21.15.27:0/2262682251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86940fd710 0x7f86941145f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:50.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:50.194+0000 7f869b800700 1 -- 172.21.15.27:0/2262682251 >> 172.21.15.27:0/2262682251 conn(0x7f86940f8540 msgr2=0x7f86940f9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:50.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:50.194+0000 7f869b800700 1 -- 172.21.15.27:0/2262682251 shutdown_connections 2024-04-16T05:39:50.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:50.194+0000 7f869b800700 1 -- 172.21.15.27:0/2262682251 wait complete. 2024-04-16T05:39:51.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:50 smithi122 ceph-mon[35998]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:51.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:50 smithi027 ceph-mon[29206]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:51.147 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-16T05:39:51.675 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:39:51.980 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:51 smithi027 ceph-mon[29206]: from='client.14422 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:39:52.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:51 smithi122 ceph-mon[35998]: from='client.14422 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:39:53.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:52 smithi122 ceph-mon[35998]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:53.028 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:52 smithi027 ceph-mon[29206]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:53.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.875+0000 7f2963000700 1 -- 172.21.15.27:0/1845647257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f295c1002b0 msgr2=0x7f295c100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:53.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.875+0000 7f2963000700 1 --2- 172.21.15.27:0/1845647257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f295c1002b0 0x7f295c100720 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2948004940 tx=0x7f2948009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:53.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.875+0000 7f2963000700 1 -- 172.21.15.27:0/1845647257 shutdown_connections 2024-04-16T05:39:53.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.875+0000 7f2963000700 1 --2- 172.21.15.27:0/1845647257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f295c1002b0 0x7f295c100720 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:53.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.875+0000 7f2963000700 1 --2- 172.21.15.27:0/1845647257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f295c0ff000 0x7f295c0ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:53.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.875+0000 7f2963000700 1 -- 172.21.15.27:0/1845647257 >> 172.21.15.27:0/1845647257 conn(0x7f295c0fa950 msgr2=0x7f295c0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:53.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.876+0000 7f2963000700 1 -- 172.21.15.27:0/1845647257 shutdown_connections 2024-04-16T05:39:53.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.876+0000 7f2963000700 1 -- 172.21.15.27:0/1845647257 wait complete. 2024-04-16T05:39:53.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.877+0000 7f2963000700 1 Processor -- start 2024-04-16T05:39:53.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.877+0000 7f2963000700 1 -- start start 2024-04-16T05:39:53.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.877+0000 7f2963000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f295c0ff000 0x7f295c10fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:53.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.877+0000 7f2963000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f295c1002b0 0x7f295c1101d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:53.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.877+0000 7f2963000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295c1107d0 con 0x7f295c1002b0 2024-04-16T05:39:53.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.877+0000 7f2963000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295c110fd0 con 0x7f295c0ff000 2024-04-16T05:39:53.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.878+0000 7f295b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f295c1002b0 0x7f295c1101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:53.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.878+0000 7f295b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f295c1002b0 0x7f295c1101d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46052/0 (socket says 172.21.15.27:46052) 2024-04-16T05:39:53.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.878+0000 7f295b400700 1 -- 172.21.15.27:0/2184094663 learned_addr learned my addr 172.21.15.27:0/2184094663 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:39:53.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.878+0000 7f295be00700 1 --2- 172.21.15.27:0/2184094663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f295c0ff000 0x7f295c10fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:53.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.878+0000 7f295b400700 1 -- 172.21.15.27:0/2184094663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f295c0ff000 msgr2=0x7f295c10fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:53.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.878+0000 7f295b400700 1 --2- 172.21.15.27:0/2184094663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f295c0ff000 0x7f295c10fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:53.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.878+0000 7f295b400700 1 -- 172.21.15.27:0/2184094663 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2948005040 con 0x7f295c1002b0 2024-04-16T05:39:53.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.879+0000 7f295b400700 1 --2- 172.21.15.27:0/2184094663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f295c1002b0 0x7f295c1101d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2948009150 tx=0x7f2948009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:53.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.879+0000 7f2953e00700 1 -- 172.21.15.27:0/2184094663 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f294800e040 con 0x7f295c1002b0 2024-04-16T05:39:53.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.879+0000 7f2953e00700 1 -- 172.21.15.27:0/2184094663 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f294800bbf0 con 0x7f295c1002b0 2024-04-16T05:39:53.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.879+0000 7f2953e00700 1 -- 172.21.15.27:0/2184094663 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f294801c630 con 0x7f295c1002b0 2024-04-16T05:39:53.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.879+0000 7f2963000700 1 -- 172.21.15.27:0/2184094663 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f295c111250 con 0x7f295c1002b0 2024-04-16T05:39:53.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.879+0000 7f2963000700 1 -- 172.21.15.27:0/2184094663 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f295c111790 con 0x7f295c1002b0 2024-04-16T05:39:53.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.880+0000 7f2963000700 1 -- 172.21.15.27:0/2184094663 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f295c05f5f0 con 0x7f295c1002b0 2024-04-16T05:39:53.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.881+0000 7f2953e00700 1 -- 172.21.15.27:0/2184094663 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f294800bd60 con 0x7f295c1002b0 2024-04-16T05:39:53.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.882+0000 7f2953e00700 1 --2- 172.21.15.27:0/2184094663 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f29400692b0 0x7f294006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:53.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.883+0000 7f2953e00700 1 -- 172.21.15.27:0/2184094663 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2948088280 con 0x7f295c1002b0 2024-04-16T05:39:53.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.883+0000 7f295be00700 1 --2- 172.21.15.27:0/2184094663 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f29400692b0 0x7f294006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:53.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.883+0000 7f295be00700 1 --2- 172.21.15.27:0/2184094663 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f29400692b0 0x7f294006b760 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f294c004990 tx=0x7f294c009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:53.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:53.888+0000 7f2953e00700 1 -- 172.21.15.27:0/2184094663 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f2948058410 con 0x7f295c1002b0 2024-04-16T05:39:54.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:54.113+0000 7f2963000700 1 -- 172.21.15.27:0/2184094663 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f295c111b60 con 0x7f29400692b0 2024-04-16T05:39:54.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:54.137+0000 7f2953e00700 1 -- 172.21.15.27:0/2184094663 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2682 (secure 0 0 0) 0x7f295c111b60 con 0x7f29400692b0 2024-04-16T05:39:54.137 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:39:54.137 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (5m) 2m ago 6m 14.1M - 0.25.0 c8568f914cd2 95a43269c2c6 2024-04-16T05:39:54.137 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (7m) 2m ago 7m 6665k - 17.2.7-847-g45317f95 b333581ccdf5 f1acb6959c63 2024-04-16T05:39:54.137 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (5m) 84s ago 5m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 e81efc20e8c0 2024-04-16T05:39:54.137 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (4m) 2m ago 6m 74.1M - 9.4.7 954c08fa6188 0de7ee73579d 2024-04-16T05:39:54.137 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:9283 running (8m) 2m ago 8m 473M - 17.2.7-847-g45317f95 b333581ccdf5 78f461c198fe 2024-04-16T05:39:54.137 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283 running (5m) 84s ago 5m 433M - 17.2.7-847-g45317f95 b333581ccdf5 1ed822ae132b 2024-04-16T05:39:54.137 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (8m) 2m ago 8m 41.5M 2048M 17.2.7-847-g45317f95 b333581ccdf5 88575dbb8d98 2024-04-16T05:39:54.137 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (5m) 84s ago 5m 37.6M 2048M 17.2.7-847-g45317f95 b333581ccdf5 5df19eab6d21 2024-04-16T05:39:54.137 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (7m) 2m ago 7m 8024k - 1.5.0 0da6a335fe13 26bcc8d81c89 2024-04-16T05:39:54.138 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (5m) 84s ago 5m 7453k - 1.5.0 0da6a335fe13 7f796c66dff6 2024-04-16T05:39:54.138 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (4m) 2m ago 4m 38.7M 4288M 17.2.7-847-g45317f95 b333581ccdf5 15e049573bca 2024-04-16T05:39:54.138 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (3m) 2m ago 3m 38.3M 4288M 17.2.7-847-g45317f95 b333581ccdf5 e146295f6500 2024-04-16T05:39:54.138 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (3m) 2m ago 3m 39.7M 4288M 17.2.7-847-g45317f95 b333581ccdf5 6212f6a18fa4 2024-04-16T05:39:54.138 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (2m) 84s ago 2m 38.2M 5312M 17.2.7-847-g45317f95 b333581ccdf5 d6724ad05f9f 2024-04-16T05:39:54.138 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (2m) 84s ago 2m 35.7M 5312M 17.2.7-847-g45317f95 b333581ccdf5 7a59908a3e10 2024-04-16T05:39:54.138 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (90s) 84s ago 89s 10.4M 5312M 17.2.7-847-g45317f95 b333581ccdf5 8e4b60163907 2024-04-16T05:39:54.138 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (4m) 2m ago 6m 38.2M - 2.43.0 a07b618ecd1d 719577e1340d 2024-04-16T05:39:54.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:54.140+0000 7f2963000700 1 -- 172.21.15.27:0/2184094663 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f29400692b0 msgr2=0x7f294006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:54.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:54.140+0000 7f2963000700 1 --2- 172.21.15.27:0/2184094663 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f29400692b0 0x7f294006b760 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f294c004990 tx=0x7f294c009ee0 comp rx=0 tx=0).stop 2024-04-16T05:39:54.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:54.140+0000 7f2963000700 1 -- 172.21.15.27:0/2184094663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f295c1002b0 msgr2=0x7f295c1101d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:54.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:54.140+0000 7f2963000700 1 --2- 172.21.15.27:0/2184094663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f295c1002b0 0x7f295c1101d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2948009150 tx=0x7f2948009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:54.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:54.140+0000 7f2963000700 1 -- 172.21.15.27:0/2184094663 shutdown_connections 2024-04-16T05:39:54.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:54.140+0000 7f2963000700 1 --2- 172.21.15.27:0/2184094663 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f29400692b0 0x7f294006b760 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:54.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:54.140+0000 7f2963000700 1 --2- 172.21.15.27:0/2184094663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f295c0ff000 0x7f295c10fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:54.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:54.140+0000 7f2963000700 1 --2- 172.21.15.27:0/2184094663 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f295c1002b0 0x7f295c1101d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:54.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:54.140+0000 7f2963000700 1 -- 172.21.15.27:0/2184094663 >> 172.21.15.27:0/2184094663 conn(0x7f295c0fa950 msgr2=0x7f295c0fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:54.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:54.141+0000 7f2963000700 1 -- 172.21.15.27:0/2184094663 shutdown_connections 2024-04-16T05:39:54.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:54.141+0000 7f2963000700 1 -- 172.21.15.27:0/2184094663 wait complete. 2024-04-16T05:39:54.769 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:54 smithi027 ceph-mon[29206]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:54.858 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-16T05:39:55.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:54 smithi122 ceph-mon[35998]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:55.383 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:39:55.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:55 smithi027 ceph-mon[29206]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:39:56.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:55 smithi122 ceph-mon[35998]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:39:57.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:56 smithi122 ceph-mon[35998]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:57.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:56 smithi027 ceph-mon[29206]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:57.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.681+0000 7feacc600700 1 -- 172.21.15.27:0/741565307 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feac41046d0 msgr2=0x7feac4104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:57.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.681+0000 7feacc600700 1 --2- 172.21.15.27:0/741565307 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feac41046d0 0x7feac4104b40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7feab0004940 tx=0x7feab0009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:57.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.681+0000 7feacc600700 1 -- 172.21.15.27:0/741565307 shutdown_connections 2024-04-16T05:39:57.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.682+0000 7feacc600700 1 --2- 172.21.15.27:0/741565307 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feac41046d0 0x7feac4104b40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:57.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.682+0000 7feacc600700 1 --2- 172.21.15.27:0/741565307 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feac4103420 0x7feac4103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.682+0000 7feacc600700 1 -- 172.21.15.27:0/741565307 >> 172.21.15.27:0/741565307 conn(0x7feac4072640 msgr2=0x7feac4074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.682+0000 7feacc600700 1 -- 172.21.15.27:0/741565307 shutdown_connections 2024-04-16T05:39:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.682+0000 7feacc600700 1 -- 172.21.15.27:0/741565307 wait complete. 2024-04-16T05:39:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.683+0000 7feacc600700 1 Processor -- start 2024-04-16T05:39:57.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.683+0000 7feacc600700 1 -- start start 2024-04-16T05:39:57.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.683+0000 7feacc600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feac4103420 0x7feac4113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:57.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.683+0000 7feacc600700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feac41046d0 0x7feac4114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:57.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.684+0000 7feacc600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac4114a90 con 0x7feac4103420 2024-04-16T05:39:57.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.684+0000 7feacc600700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac4114c00 con 0x7feac41046d0 2024-04-16T05:39:57.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.684+0000 7feac9c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feac4103420 0x7feac4113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:57.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.684+0000 7feac9c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feac4103420 0x7feac4113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50920/0 (socket says 172.21.15.27:50920) 2024-04-16T05:39:57.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.684+0000 7feac9c00700 1 -- 172.21.15.27:0/3570074461 learned_addr learned my addr 172.21.15.27:0/3570074461 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:39:57.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.684+0000 7feac9200700 1 --2- 172.21.15.27:0/3570074461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feac41046d0 0x7feac4114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:57.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.684+0000 7feac9c00700 1 -- 172.21.15.27:0/3570074461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feac41046d0 msgr2=0x7feac4114490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:57.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.684+0000 7feac9c00700 1 --2- 172.21.15.27:0/3570074461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feac41046d0 0x7feac4114490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:57.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.684+0000 7feac9c00700 1 -- 172.21.15.27:0/3570074461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab0005040 con 0x7feac4103420 2024-04-16T05:39:57.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.685+0000 7feac9c00700 1 --2- 172.21.15.27:0/3570074461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feac4103420 0x7feac4113f50 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7feab8002a50 tx=0x7feab80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:57.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.685+0000 7feac1600700 1 -- 172.21.15.27:0/3570074461 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feab8007cc0 con 0x7feac4103420 2024-04-16T05:39:57.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.685+0000 7feac1600700 1 -- 172.21.15.27:0/3570074461 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feab800f040 con 0x7feac4103420 2024-04-16T05:39:57.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.685+0000 7feac1600700 1 -- 172.21.15.27:0/3570074461 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feab800db70 con 0x7feac4103420 2024-04-16T05:39:57.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.685+0000 7feacc600700 1 -- 172.21.15.27:0/3570074461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac4119660 con 0x7feac4103420 2024-04-16T05:39:57.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.685+0000 7feacc600700 1 -- 172.21.15.27:0/3570074461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac4119c20 con 0x7feac4103420 2024-04-16T05:39:57.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.687+0000 7feacc600700 1 -- 172.21.15.27:0/3570074461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac404c710 con 0x7feac4103420 2024-04-16T05:39:57.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.687+0000 7feac1600700 1 -- 172.21.15.27:0/3570074461 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7feab800dd10 con 0x7feac4103420 2024-04-16T05:39:57.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.688+0000 7feac1600700 1 --2- 172.21.15.27:0/3570074461 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7feaac0692a0 0x7feaac06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:39:57.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.688+0000 7feac1600700 1 -- 172.21.15.27:0/3570074461 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7feab8086560 con 0x7feac4103420 2024-04-16T05:39:57.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.689+0000 7feac9200700 1 --2- 172.21.15.27:0/3570074461 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7feaac0692a0 0x7feaac06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:39:57.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.689+0000 7feac9200700 1 --2- 172.21.15.27:0/3570074461 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7feaac0692a0 0x7feaac06b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7feab0004f90 tx=0x7feab0009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:39:57.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.693+0000 7feac1600700 1 -- 172.21.15.27:0/3570074461 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7feab80565c0 con 0x7feac4103420 2024-04-16T05:39:57.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.920+0000 7feacc600700 1 -- 172.21.15.27:0/3570074461 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7feac41089f0 con 0x7feaac0692a0 2024-04-16T05:39:57.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.932+0000 7feac1600700 1 -- 172.21.15.27:0/3570074461 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1170 (secure 0 0 0) 0x7feac41089f0 con 0x7feaac0692a0 2024-04-16T05:39:57.931 INFO:teuthology.orchestra.run.smithi027.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-16T05:39:57.932 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager ?:9093,9094 1/1 2m ago 7m count:1 2024-04-16T05:39:57.932 INFO:teuthology.orchestra.run.smithi027.stdout:crash 2/2 2m ago 8m * 2024-04-16T05:39:57.932 INFO:teuthology.orchestra.run.smithi027.stdout:grafana ?:3000 1/1 2m ago 7m count:1 2024-04-16T05:39:57.932 INFO:teuthology.orchestra.run.smithi027.stdout:mgr 2/2 2m ago 8m count:2 2024-04-16T05:39:57.932 INFO:teuthology.orchestra.run.smithi027.stdout:mon 2/2 2m ago 6m smithi027:172.21.15.27=smithi027;smithi122:172.21.15.122=smithi122;count:2 2024-04-16T05:39:57.932 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter ?:9100 2/2 2m ago 7m * 2024-04-16T05:39:57.932 INFO:teuthology.orchestra.run.smithi027.stdout:osd 6 2m ago - 2024-04-16T05:39:57.932 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus ?:9095 1/1 2m ago 8m count:1 2024-04-16T05:39:57.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.935+0000 7feacc600700 1 -- 172.21.15.27:0/3570074461 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7feaac0692a0 msgr2=0x7feaac06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:57.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.935+0000 7feacc600700 1 --2- 172.21.15.27:0/3570074461 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7feaac0692a0 0x7feaac06b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7feab0004f90 tx=0x7feab0009e30 comp rx=0 tx=0).stop 2024-04-16T05:39:57.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.935+0000 7feacc600700 1 -- 172.21.15.27:0/3570074461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feac4103420 msgr2=0x7feac4113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:39:57.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.935+0000 7feacc600700 1 --2- 172.21.15.27:0/3570074461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feac4103420 0x7feac4113f50 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7feab8002a50 tx=0x7feab80046b0 comp rx=0 tx=0).stop 2024-04-16T05:39:57.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.935+0000 7feacc600700 1 -- 172.21.15.27:0/3570074461 shutdown_connections 2024-04-16T05:39:57.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.935+0000 7feacc600700 1 --2- 172.21.15.27:0/3570074461 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7feaac0692a0 0x7feaac06b750 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:57.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.935+0000 7feacc600700 1 --2- 172.21.15.27:0/3570074461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feac4103420 0x7feac4113f50 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:57.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.935+0000 7feacc600700 1 --2- 172.21.15.27:0/3570074461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feac41046d0 0x7feac4114490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:39:57.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.935+0000 7feacc600700 1 -- 172.21.15.27:0/3570074461 >> 172.21.15.27:0/3570074461 conn(0x7feac4072640 msgr2=0x7feac4073d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:39:57.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.936+0000 7feacc600700 1 -- 172.21.15.27:0/3570074461 shutdown_connections 2024-04-16T05:39:57.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:39:57.936+0000 7feacc600700 1 -- 172.21.15.27:0/3570074461 wait complete. 2024-04-16T05:39:58.758 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-16T05:39:59.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:58 smithi122 ceph-mon[35998]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:59.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:58 smithi027 ceph-mon[29206]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:39:59.269 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:40:00.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:39:59 smithi122 ceph-mon[35998]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:40:00.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:39:59 smithi027 ceph-mon[29206]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:40:01.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:00 smithi122 ceph-mon[35998]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:01.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:00 smithi122 ceph-mon[35998]: overall HEALTH_OK 2024-04-16T05:40:01.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:00 smithi027 ceph-mon[29206]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:01.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:00 smithi027 ceph-mon[29206]: overall HEALTH_OK 2024-04-16T05:40:01.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.676+0000 7fd687e00700 1 -- 172.21.15.27:0/4034171184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6800fcff0 msgr2=0x7fd6800fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:01.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.676+0000 7fd687e00700 1 --2- 172.21.15.27:0/4034171184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6800fcff0 0x7fd6800fd460 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd66c004940 tx=0x7fd66c009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:01.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.676+0000 7fd687e00700 1 -- 172.21.15.27:0/4034171184 shutdown_connections 2024-04-16T05:40:01.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.676+0000 7fd687e00700 1 --2- 172.21.15.27:0/4034171184 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6800fcff0 0x7fd6800fd460 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:01.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.676+0000 7fd687e00700 1 --2- 172.21.15.27:0/4034171184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6800fbd40 0x7fd6800fc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:01.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.676+0000 7fd687e00700 1 -- 172.21.15.27:0/4034171184 >> 172.21.15.27:0/4034171184 conn(0x7fd6800f74c0 msgr2=0x7fd6800f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:01.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.677+0000 7fd687e00700 1 -- 172.21.15.27:0/4034171184 shutdown_connections 2024-04-16T05:40:01.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.677+0000 7fd687e00700 1 -- 172.21.15.27:0/4034171184 wait complete. 2024-04-16T05:40:01.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.677+0000 7fd687e00700 1 Processor -- start 2024-04-16T05:40:01.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.678+0000 7fd687e00700 1 -- start start 2024-04-16T05:40:01.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.678+0000 7fd687e00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6800fbd40 0x7fd68010fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:01.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.678+0000 7fd687e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6800fcff0 0x7fd680110170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:01.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.678+0000 7fd687e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd680110770 con 0x7fd6800fcff0 2024-04-16T05:40:01.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.678+0000 7fd687e00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd680110fc0 con 0x7fd6800fbd40 2024-04-16T05:40:01.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.678+0000 7fd684a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6800fcff0 0x7fd680110170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:01.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.678+0000 7fd684a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6800fcff0 0x7fd680110170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50938/0 (socket says 172.21.15.27:50938) 2024-04-16T05:40:01.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.678+0000 7fd684a00700 1 -- 172.21.15.27:0/496919344 learned_addr learned my addr 172.21.15.27:0/496919344 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:40:01.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.678+0000 7fd685400700 1 --2- 172.21.15.27:0/496919344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6800fbd40 0x7fd68010fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:01.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.679+0000 7fd684a00700 1 -- 172.21.15.27:0/496919344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6800fbd40 msgr2=0x7fd68010fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:01.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.679+0000 7fd684a00700 1 --2- 172.21.15.27:0/496919344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6800fbd40 0x7fd68010fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:01.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.679+0000 7fd684a00700 1 -- 172.21.15.27:0/496919344 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c005040 con 0x7fd6800fcff0 2024-04-16T05:40:01.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.679+0000 7fd684a00700 1 --2- 172.21.15.27:0/496919344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6800fcff0 0x7fd680110170 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd66c009150 tx=0x7fd66c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:01.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.679+0000 7fd67d600700 1 -- 172.21.15.27:0/496919344 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd66c00e040 con 0x7fd6800fcff0 2024-04-16T05:40:01.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.679+0000 7fd67d600700 1 -- 172.21.15.27:0/496919344 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd66c00bcc0 con 0x7fd6800fcff0 2024-04-16T05:40:01.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.679+0000 7fd687e00700 1 -- 172.21.15.27:0/496919344 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd680111240 con 0x7fd6800fcff0 2024-04-16T05:40:01.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.679+0000 7fd67d600700 1 -- 172.21.15.27:0/496919344 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd66c01c630 con 0x7fd6800fcff0 2024-04-16T05:40:01.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.679+0000 7fd687e00700 1 -- 172.21.15.27:0/496919344 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd680111780 con 0x7fd6800fcff0 2024-04-16T05:40:01.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.681+0000 7fd687e00700 1 -- 172.21.15.27:0/496919344 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd680101980 con 0x7fd6800fcff0 2024-04-16T05:40:01.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.682+0000 7fd67d600700 1 -- 172.21.15.27:0/496919344 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fd66c013420 con 0x7fd6800fcff0 2024-04-16T05:40:01.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.683+0000 7fd67d600700 1 --2- 172.21.15.27:0/496919344 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd6680692a0 0x7fd66806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:01.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.683+0000 7fd67d600700 1 -- 172.21.15.27:0/496919344 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd66c087c00 con 0x7fd6800fcff0 2024-04-16T05:40:01.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.683+0000 7fd685400700 1 --2- 172.21.15.27:0/496919344 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd6680692a0 0x7fd66806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:01.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.684+0000 7fd685400700 1 --2- 172.21.15.27:0/496919344 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd6680692a0 0x7fd66806b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd674004990 tx=0x7fd674009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:01.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.688+0000 7fd67d600700 1 -- 172.21.15.27:0/496919344 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fd66c057ce0 con 0x7fd6800fcff0 2024-04-16T05:40:01.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.915+0000 7fd687e00700 1 -- 172.21.15.27:0/496919344 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd680104980 con 0x7fd6680692a0 2024-04-16T05:40:01.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.918+0000 7fd67d600700 1 -- 172.21.15.27:0/496919344 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7fd680104980 con 0x7fd6680692a0 2024-04-16T05:40:01.918 INFO:teuthology.orchestra.run.smithi027.stdout:HOST ADDR LABELS STATUS 2024-04-16T05:40:01.918 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 172.21.15.27 2024-04-16T05:40:01.918 INFO:teuthology.orchestra.run.smithi027.stdout:smithi122 172.21.15.122 2024-04-16T05:40:01.918 INFO:teuthology.orchestra.run.smithi027.stdout:2 hosts in cluster 2024-04-16T05:40:01.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.921+0000 7fd687e00700 1 -- 172.21.15.27:0/496919344 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd6680692a0 msgr2=0x7fd66806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:01.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.921+0000 7fd687e00700 1 --2- 172.21.15.27:0/496919344 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd6680692a0 0x7fd66806b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd674004990 tx=0x7fd674009ee0 comp rx=0 tx=0).stop 2024-04-16T05:40:01.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.921+0000 7fd687e00700 1 -- 172.21.15.27:0/496919344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6800fcff0 msgr2=0x7fd680110170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:01.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.921+0000 7fd687e00700 1 --2- 172.21.15.27:0/496919344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6800fcff0 0x7fd680110170 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd66c009150 tx=0x7fd66c009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:01.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.921+0000 7fd687e00700 1 -- 172.21.15.27:0/496919344 shutdown_connections 2024-04-16T05:40:01.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.921+0000 7fd687e00700 1 --2- 172.21.15.27:0/496919344 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd6680692a0 0x7fd66806b750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:01.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.921+0000 7fd687e00700 1 --2- 172.21.15.27:0/496919344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6800fbd40 0x7fd68010fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:01.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.921+0000 7fd687e00700 1 --2- 172.21.15.27:0/496919344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6800fcff0 0x7fd680110170 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:01.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.921+0000 7fd687e00700 1 -- 172.21.15.27:0/496919344 >> 172.21.15.27:0/496919344 conn(0x7fd6800f74c0 msgr2=0x7fd680065b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:01.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.922+0000 7fd687e00700 1 -- 172.21.15.27:0/496919344 shutdown_connections 2024-04-16T05:40:01.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:01.922+0000 7fd687e00700 1 -- 172.21.15.27:0/496919344 wait complete. 2024-04-16T05:40:02.844 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-16T05:40:03.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:02 smithi122 ceph-mon[35998]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:03.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:02 smithi027 ceph-mon[29206]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:03.362 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:40:03.780 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:03 smithi027 ceph-mon[29206]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:40:04.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:03 smithi122 ceph-mon[35998]: from='client.14434 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:40:05.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:04 smithi122 ceph-mon[35998]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:05.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:04 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:40:05.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:04 smithi027 ceph-mon[29206]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:05.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:04 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:40:05.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.746+0000 7f7312000700 1 -- 172.21.15.27:0/3349899786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c0fce80 msgr2=0x7f730c0fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:05.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.746+0000 7f7312000700 1 --2- 172.21.15.27:0/3349899786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c0fce80 0x7f730c0fd290 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f72fc004940 tx=0x7f72fc009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:05.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.746+0000 7f7312000700 1 -- 172.21.15.27:0/3349899786 shutdown_connections 2024-04-16T05:40:05.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.746+0000 7f7312000700 1 --2- 172.21.15.27:0/3349899786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f730c0fdff0 0x7f730c0fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:05.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.746+0000 7f7312000700 1 --2- 172.21.15.27:0/3349899786 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c0fce80 0x7f730c0fd290 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:05.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.746+0000 7f7312000700 1 -- 172.21.15.27:0/3349899786 >> 172.21.15.27:0/3349899786 conn(0x7f730c0f8540 msgr2=0x7f730c0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:05.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.746+0000 7f7312000700 1 -- 172.21.15.27:0/3349899786 shutdown_connections 2024-04-16T05:40:05.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.747+0000 7f7312000700 1 -- 172.21.15.27:0/3349899786 wait complete. 2024-04-16T05:40:05.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.747+0000 7f7312000700 1 Processor -- start 2024-04-16T05:40:05.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.748+0000 7f7312000700 1 -- start start 2024-04-16T05:40:05.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.748+0000 7f7312000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f730c0fce80 0x7f730c10fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:05.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.748+0000 7f7312000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c0fdff0 0x7f730c1100e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:05.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.748+0000 7f7312000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c1106e0 con 0x7f730c0fdff0 2024-04-16T05:40:05.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.748+0000 7f7312000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c110850 con 0x7f730c0fce80 2024-04-16T05:40:05.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.748+0000 7f730aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c0fdff0 0x7f730c1100e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:05.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.748+0000 7f730aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c0fdff0 0x7f730c1100e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37364/0 (socket says 172.21.15.27:37364) 2024-04-16T05:40:05.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.748+0000 7f730aa00700 1 -- 172.21.15.27:0/116692098 learned_addr learned my addr 172.21.15.27:0/116692098 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:40:05.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.748+0000 7f730b400700 1 --2- 172.21.15.27:0/116692098 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f730c0fce80 0x7f730c10fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:05.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.749+0000 7f730aa00700 1 -- 172.21.15.27:0/116692098 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f730c0fce80 msgr2=0x7f730c10fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:05.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.749+0000 7f730aa00700 1 --2- 172.21.15.27:0/116692098 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f730c0fce80 0x7f730c10fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:05.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.749+0000 7f730aa00700 1 -- 172.21.15.27:0/116692098 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc005040 con 0x7f730c0fdff0 2024-04-16T05:40:05.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.749+0000 7f730b400700 1 --2- 172.21.15.27:0/116692098 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f730c0fce80 0x7f730c10fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T05:40:05.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.749+0000 7f730aa00700 1 --2- 172.21.15.27:0/116692098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c0fdff0 0x7f730c1100e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f72f800c830 tx=0x7f72f8004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:05.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.749+0000 7f7303400700 1 -- 172.21.15.27:0/116692098 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72f800a760 con 0x7f730c0fdff0 2024-04-16T05:40:05.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.750+0000 7f7303400700 1 -- 172.21.15.27:0/116692098 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f72f800ad90 con 0x7f730c0fdff0 2024-04-16T05:40:05.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.750+0000 7f7303400700 1 -- 172.21.15.27:0/116692098 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72f80113a0 con 0x7f730c0fdff0 2024-04-16T05:40:05.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.750+0000 7f7312000700 1 -- 172.21.15.27:0/116692098 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f730c1152b0 con 0x7f730c0fdff0 2024-04-16T05:40:05.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.750+0000 7f7312000700 1 -- 172.21.15.27:0/116692098 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730c115820 con 0x7f730c0fdff0 2024-04-16T05:40:05.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.751+0000 7f7312000700 1 -- 172.21.15.27:0/116692098 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f730c101980 con 0x7f730c0fdff0 2024-04-16T05:40:05.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.752+0000 7f7303400700 1 -- 172.21.15.27:0/116692098 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f72f8012400 con 0x7f730c0fdff0 2024-04-16T05:40:05.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.753+0000 7f7303400700 1 --2- 172.21.15.27:0/116692098 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f72f00692a0 0x7f72f006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:05.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.753+0000 7f7303400700 1 -- 172.21.15.27:0/116692098 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f72f8086fa0 con 0x7f730c0fdff0 2024-04-16T05:40:05.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.753+0000 7f730b400700 1 --2- 172.21.15.27:0/116692098 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f72f00692a0 0x7f72f006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:05.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.754+0000 7f730b400700 1 --2- 172.21.15.27:0/116692098 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f72f00692a0 0x7f72f006b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f72fc00b020 tx=0x7f72fc009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:05.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.761+0000 7f7303400700 1 -- 172.21.15.27:0/116692098 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f72f8057000 con 0x7f730c0fdff0 2024-04-16T05:40:05.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.987+0000 7f7312000700 1 -- 172.21.15.27:0/116692098 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f730c02b3b0 con 0x7f72f00692a0 2024-04-16T05:40:05.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.991+0000 7f7303400700 1 -- 172.21.15.27:0/116692098 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f730c02b3b0 con 0x7f72f00692a0 2024-04-16T05:40:05.990 INFO:teuthology.orchestra.run.smithi027.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-16T05:40:05.990 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5331001C400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-16T05:40:05.990 INFO:teuthology.orchestra.run.smithi027.stdout:smithi122 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204010U400BGN 372G No 91s ago Has a FileSystem, LVM detected 2024-04-16T05:40:05.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.994+0000 7f7312000700 1 -- 172.21.15.27:0/116692098 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f72f00692a0 msgr2=0x7f72f006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.994+0000 7f7312000700 1 --2- 172.21.15.27:0/116692098 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f72f00692a0 0x7f72f006b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f72fc00b020 tx=0x7f72fc009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.994+0000 7f7312000700 1 -- 172.21.15.27:0/116692098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c0fdff0 msgr2=0x7f730c1100e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.994+0000 7f7312000700 1 --2- 172.21.15.27:0/116692098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c0fdff0 0x7f730c1100e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f72f800c830 tx=0x7f72f8004470 comp rx=0 tx=0).stop 2024-04-16T05:40:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.994+0000 7f7312000700 1 -- 172.21.15.27:0/116692098 shutdown_connections 2024-04-16T05:40:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.994+0000 7f7312000700 1 --2- 172.21.15.27:0/116692098 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f72f00692a0 0x7f72f006b750 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.994+0000 7f7312000700 1 --2- 172.21.15.27:0/116692098 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f730c0fce80 0x7f730c10fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.994+0000 7f7312000700 1 --2- 172.21.15.27:0/116692098 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f730c0fdff0 0x7f730c1100e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.994+0000 7f7312000700 1 -- 172.21.15.27:0/116692098 >> 172.21.15.27:0/116692098 conn(0x7f730c0f8540 msgr2=0x7f730c0f9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.994+0000 7f7312000700 1 -- 172.21.15.27:0/116692098 shutdown_connections 2024-04-16T05:40:05.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:05.994+0000 7f7312000700 1 -- 172.21.15.27:0/116692098 wait complete. 2024-04-16T05:40:06.809 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T05:40:06.827 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-16T05:40:06.827 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-04-16T05:40:07.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:06 smithi122 ceph-mon[35998]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:07.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:06 smithi027 ceph-mon[29206]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:07.327 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:40:08.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:07 smithi122 ceph-mon[35998]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:40:08.029 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:07 smithi027 ceph-mon[29206]: from='client.14438 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:40:09.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:08 smithi122 ceph-mon[35998]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:09.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:08 smithi027 ceph-mon[29206]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:09.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.756+0000 7f01f7600700 1 -- 172.21.15.27:0/2932611489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01f00ff020 msgr2=0x7f01f00ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:09.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.756+0000 7f01f7600700 1 --2- 172.21.15.27:0/2932611489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01f00ff020 0x7f01f00ff430 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f01e4004940 tx=0x7f01e4009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:09.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.756+0000 7f01f7600700 1 -- 172.21.15.27:0/2932611489 shutdown_connections 2024-04-16T05:40:09.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.756+0000 7f01f7600700 1 --2- 172.21.15.27:0/2932611489 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01f01002d0 0x7f01f0100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:09.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.756+0000 7f01f7600700 1 --2- 172.21.15.27:0/2932611489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01f00ff020 0x7f01f00ff430 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:09.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.756+0000 7f01f7600700 1 -- 172.21.15.27:0/2932611489 >> 172.21.15.27:0/2932611489 conn(0x7f01f00fa950 msgr2=0x7f01f00fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:09.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.756+0000 7f01f7600700 1 -- 172.21.15.27:0/2932611489 shutdown_connections 2024-04-16T05:40:09.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.757+0000 7f01f7600700 1 -- 172.21.15.27:0/2932611489 wait complete. 2024-04-16T05:40:09.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.758+0000 7f01f7600700 1 Processor -- start 2024-04-16T05:40:09.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.758+0000 7f01f7600700 1 -- start start 2024-04-16T05:40:09.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.759+0000 7f01f7600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01f00ff020 0x7f01f010fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:09.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.759+0000 7f01f7600700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01f01002d0 0x7f01f0110070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:09.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.759+0000 7f01f7600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f0110700 con 0x7f01f00ff020 2024-04-16T05:40:09.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.759+0000 7f01f7600700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f01150a0 con 0x7f01f01002d0 2024-04-16T05:40:09.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.759+0000 7f01f4c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01f00ff020 0x7f01f010fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:09.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.759+0000 7f01f4c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01f00ff020 0x7f01f010fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37390/0 (socket says 172.21.15.27:37390) 2024-04-16T05:40:09.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.759+0000 7f01f4c00700 1 -- 172.21.15.27:0/993528845 learned_addr learned my addr 172.21.15.27:0/993528845 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:40:09.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.760+0000 7f01efe00700 1 --2- 172.21.15.27:0/993528845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01f01002d0 0x7f01f0110070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:09.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.760+0000 7f01f4c00700 1 -- 172.21.15.27:0/993528845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01f01002d0 msgr2=0x7f01f0110070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:09.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.760+0000 7f01f4c00700 1 --2- 172.21.15.27:0/993528845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01f01002d0 0x7f01f0110070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:09.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.760+0000 7f01f4c00700 1 -- 172.21.15.27:0/993528845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e4005040 con 0x7f01f00ff020 2024-04-16T05:40:09.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.760+0000 7f01f4c00700 1 --2- 172.21.15.27:0/993528845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01f00ff020 0x7f01f010fb30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f01e4009150 tx=0x7f01e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:09.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.760+0000 7f01ecc00700 1 -- 172.21.15.27:0/993528845 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01e400e040 con 0x7f01f00ff020 2024-04-16T05:40:09.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.761+0000 7f01f7600700 1 -- 172.21.15.27:0/993528845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f01152a0 con 0x7f01f00ff020 2024-04-16T05:40:09.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.761+0000 7f01ecc00700 1 -- 172.21.15.27:0/993528845 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01e400bcc0 con 0x7f01f00ff020 2024-04-16T05:40:09.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.761+0000 7f01ecc00700 1 -- 172.21.15.27:0/993528845 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01e401c630 con 0x7f01f00ff020 2024-04-16T05:40:09.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.761+0000 7f01f7600700 1 -- 172.21.15.27:0/993528845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f0115810 con 0x7f01f00ff020 2024-04-16T05:40:09.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.762+0000 7f01f7600700 1 -- 172.21.15.27:0/993528845 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f004c710 con 0x7f01f00ff020 2024-04-16T05:40:09.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.763+0000 7f01ecc00700 1 -- 172.21.15.27:0/993528845 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f01e4013420 con 0x7f01f00ff020 2024-04-16T05:40:09.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.764+0000 7f01ecc00700 1 --2- 172.21.15.27:0/993528845 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f01d80692a0 0x7f01d806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:09.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.764+0000 7f01ecc00700 1 -- 172.21.15.27:0/993528845 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f01e4087bf0 con 0x7f01f00ff020 2024-04-16T05:40:09.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.764+0000 7f01efe00700 1 --2- 172.21.15.27:0/993528845 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f01d80692a0 0x7f01d806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:09.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.765+0000 7f01efe00700 1 --2- 172.21.15.27:0/993528845 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f01d80692a0 0x7f01d806b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f01dc006fd0 tx=0x7f01dc008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:09.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:09.769+0000 7f01ecc00700 1 -- 172.21.15.27:0/993528845 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f01e4057cd0 con 0x7f01f00ff020 2024-04-16T05:40:10.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:10.017+0000 7f01f7600700 1 -- 172.21.15.27:0/993528845 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f01f0115a30 con 0x7f01d80692a0 2024-04-16T05:40:11.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:10 smithi027 ceph-mon[29206]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:11.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:10 smithi122 ceph-mon[35998]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:11.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:11.893+0000 7f01ecc00700 1 -- 172.21.15.27:0/993528845 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f01f0115a30 con 0x7f01d80692a0 2024-04-16T05:40:11.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:11.898+0000 7f01f7600700 1 -- 172.21.15.27:0/993528845 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f01d80692a0 msgr2=0x7f01d806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:11.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:11.898+0000 7f01f7600700 1 --2- 172.21.15.27:0/993528845 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f01d80692a0 0x7f01d806b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f01dc006fd0 tx=0x7f01dc008040 comp rx=0 tx=0).stop 2024-04-16T05:40:11.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:11.898+0000 7f01f7600700 1 -- 172.21.15.27:0/993528845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01f00ff020 msgr2=0x7f01f010fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:11.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:11.898+0000 7f01f7600700 1 --2- 172.21.15.27:0/993528845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01f00ff020 0x7f01f010fb30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f01e4009150 tx=0x7f01e4009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:11.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:11.898+0000 7f01f7600700 1 -- 172.21.15.27:0/993528845 shutdown_connections 2024-04-16T05:40:11.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:11.898+0000 7f01f7600700 1 --2- 172.21.15.27:0/993528845 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f01d80692a0 0x7f01d806b750 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:11.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:11.898+0000 7f01f7600700 1 --2- 172.21.15.27:0/993528845 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f01f00ff020 0x7f01f010fb30 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:11.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:11.898+0000 7f01f7600700 1 --2- 172.21.15.27:0/993528845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f01f01002d0 0x7f01f0110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:11.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:11.898+0000 7f01f7600700 1 -- 172.21.15.27:0/993528845 >> 172.21.15.27:0/993528845 conn(0x7f01f00fa950 msgr2=0x7f01f00fcf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:11.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:11.899+0000 7f01f7600700 1 -- 172.21.15.27:0/993528845 shutdown_connections 2024-04-16T05:40:11.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:11.899+0000 7f01f7600700 1 -- 172.21.15.27:0/993528845 wait complete. 2024-04-16T05:40:12.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:11 smithi122 ceph-mon[35998]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:40:12.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:11 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-16T05:40:12.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:11 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-16T05:40:12.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:11 smithi122 ceph-mon[35998]: osdmap e41: 6 total, 6 up, 6 in 2024-04-16T05:40:12.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:11 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-16T05:40:12.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:11 smithi122 ceph-mon[35998]: pgmap v220: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:12.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:11 smithi027 ceph-mon[29206]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:40:12.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:11 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-04-16T05:40:12.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:11 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-04-16T05:40:12.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:11 smithi027 ceph-mon[29206]: osdmap e41: 6 total, 6 up, 6 in 2024-04-16T05:40:12.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:11 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-04-16T05:40:12.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:11 smithi027 ceph-mon[29206]: pgmap v220: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:12.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:11 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027[29201]: 2024-04-16T05:40:11.858+0000 7fad0d600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T05:40:12.732 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-16T05:40:12.908 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:12 smithi027 ceph-mon[29206]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-16T05:40:12.909 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:12 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-16T05:40:12.909 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:12 smithi027 ceph-mon[29206]: osdmap e42: 6 total, 6 up, 6 in 2024-04-16T05:40:12.909 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:12 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-16T05:40:12.909 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:12 smithi027 ceph-mon[29206]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T05:40:12.909 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:12 smithi027 ceph-mon[29206]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-16T05:40:12.909 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:12 smithi027 ceph-mon[29206]: osdmap e43: 6 total, 6 up, 6 in 2024-04-16T05:40:12.909 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:12 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-16T05:40:12.909 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:12 smithi027 ceph-mon[29206]: fsmap cephfs:0 2024-04-16T05:40:12.909 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:12 smithi027 ceph-mon[29206]: Saving service mds.cephfs spec with placement count:4 2024-04-16T05:40:12.909 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:12 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:12.909 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:12 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:40:12.909 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:12 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:40:12.910 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:12 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:40:12.910 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:12 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:12.910 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:12 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.hkhpuq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:40:12.910 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:12 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.hkhpuq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T05:40:12.910 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:12 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:40:12.910 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:12 smithi027 ceph-mon[29206]: Deploying daemon mds.cephfs.smithi027.hkhpuq on smithi027 2024-04-16T05:40:13.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:12 smithi122 ceph-mon[35998]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-16T05:40:13.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:12 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-04-16T05:40:13.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:12 smithi122 ceph-mon[35998]: osdmap e42: 6 total, 6 up, 6 in 2024-04-16T05:40:13.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:12 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-04-16T05:40:13.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:12 smithi122 ceph-mon[35998]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T05:40:13.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:12 smithi122 ceph-mon[35998]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-16T05:40:13.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:12 smithi122 ceph-mon[35998]: osdmap e43: 6 total, 6 up, 6 in 2024-04-16T05:40:13.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:12 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-04-16T05:40:13.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:12 smithi122 ceph-mon[35998]: fsmap cephfs:0 2024-04-16T05:40:13.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:12 smithi122 ceph-mon[35998]: Saving service mds.cephfs spec with placement count:4 2024-04-16T05:40:13.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:12 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:13.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:12 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:40:13.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:12 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:40:13.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:12 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:40:13.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:12 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:13.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:12 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.hkhpuq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:40:13.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:12 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.hkhpuq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T05:40:13.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:12 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:40:13.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:12 smithi122 ceph-mon[35998]: Deploying daemon mds.cephfs.smithi027.hkhpuq on smithi027 2024-04-16T05:40:14.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:13 smithi027 ceph-mon[29206]: osdmap e44: 6 total, 6 up, 6 in 2024-04-16T05:40:14.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:13 smithi027 ceph-mon[29206]: pgmap v224: 65 pgs: 17 creating+peering, 7 active+clean, 41 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:14.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:13 smithi122 ceph-mon[35998]: osdmap e44: 6 total, 6 up, 6 in 2024-04-16T05:40:14.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:13 smithi122 ceph-mon[35998]: pgmap v224: 65 pgs: 17 creating+peering, 7 active+clean, 41 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:14.337 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:40:15.226 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:14 smithi027 ceph-mon[29206]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-16T05:40:15.227 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:14 smithi027 ceph-mon[29206]: osdmap e45: 6 total, 6 up, 6 in 2024-04-16T05:40:15.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:14 smithi122 ceph-mon[35998]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-16T05:40:15.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:14 smithi122 ceph-mon[35998]: osdmap e45: 6 total, 6 up, 6 in 2024-04-16T05:40:16.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:15 smithi122 ceph-mon[35998]: pgmap v226: 65 pgs: 17 creating+peering, 22 active+clean, 26 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:16.304 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:15 smithi027 ceph-mon[29206]: pgmap v226: 65 pgs: 17 creating+peering, 22 active+clean, 26 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.243+0000 7fb05f400700 1 -- 172.21.15.27:0/881043709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0580fd710 msgr2=0x7fb0580fe070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.243+0000 7fb05f400700 1 --2- 172.21.15.27:0/881043709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0580fd710 0x7fb0580fe070 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb044004940 tx=0x7fb044009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.244+0000 7fb05f400700 1 -- 172.21.15.27:0/881043709 shutdown_connections 2024-04-16T05:40:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.244+0000 7fb05f400700 1 --2- 172.21.15.27:0/881043709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0580fd710 0x7fb0580fe070 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.244+0000 7fb05f400700 1 --2- 172.21.15.27:0/881043709 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0580fcdc0 0x7fb0580fd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.244+0000 7fb05f400700 1 -- 172.21.15.27:0/881043709 >> 172.21.15.27:0/881043709 conn(0x7fb0580f8540 msgr2=0x7fb0580fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.244+0000 7fb05f400700 1 -- 172.21.15.27:0/881043709 shutdown_connections 2024-04-16T05:40:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.244+0000 7fb05f400700 1 -- 172.21.15.27:0/881043709 wait complete. 2024-04-16T05:40:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.245+0000 7fb05f400700 1 Processor -- start 2024-04-16T05:40:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.245+0000 7fb05f400700 1 -- start start 2024-04-16T05:40:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.246+0000 7fb05f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0580fcdc0 0x7fb05810da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.246+0000 7fb05f400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0580fd710 0x7fb05810df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.246+0000 7fb05f400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05810e570 con 0x7fb0580fcdc0 2024-04-16T05:40:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.246+0000 7fb05f400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05810edc0 con 0x7fb0580fd710 2024-04-16T05:40:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.246+0000 7fb05ca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0580fcdc0 0x7fb05810da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:17.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.246+0000 7fb05ca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0580fcdc0 0x7fb05810da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35168/0 (socket says 172.21.15.27:35168) 2024-04-16T05:40:17.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.246+0000 7fb05ca00700 1 -- 172.21.15.27:0/633923326 learned_addr learned my addr 172.21.15.27:0/633923326 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:40:17.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.246+0000 7fb057e00700 1 --2- 172.21.15.27:0/633923326 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0580fd710 0x7fb05810df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:17.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.247+0000 7fb05ca00700 1 -- 172.21.15.27:0/633923326 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0580fd710 msgr2=0x7fb05810df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:17.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.247+0000 7fb05ca00700 1 --2- 172.21.15.27:0/633923326 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0580fd710 0x7fb05810df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:17.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.247+0000 7fb05ca00700 1 -- 172.21.15.27:0/633923326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb044005040 con 0x7fb0580fcdc0 2024-04-16T05:40:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.247+0000 7fb05ca00700 1 --2- 172.21.15.27:0/633923326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0580fcdc0 0x7fb05810da30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb04c00eb40 tx=0x7fb04c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.247+0000 7fb054c00700 1 -- 172.21.15.27:0/633923326 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb04c007cc0 con 0x7fb0580fcdc0 2024-04-16T05:40:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.247+0000 7fb054c00700 1 -- 172.21.15.27:0/633923326 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb04c00f040 con 0x7fb0580fcdc0 2024-04-16T05:40:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.248+0000 7fb054c00700 1 -- 172.21.15.27:0/633923326 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb04c00caa0 con 0x7fb0580fcdc0 2024-04-16T05:40:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.248+0000 7fb05f400700 1 -- 172.21.15.27:0/633923326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb05810f040 con 0x7fb0580fcdc0 2024-04-16T05:40:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.248+0000 7fb05f400700 1 -- 172.21.15.27:0/633923326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb05810f580 con 0x7fb0580fcdc0 2024-04-16T05:40:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.249+0000 7fb05f400700 1 -- 172.21.15.27:0/633923326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb05805f5f0 con 0x7fb0580fcdc0 2024-04-16T05:40:17.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.250+0000 7fb054c00700 1 -- 172.21.15.27:0/633923326 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fb04c00cc00 con 0x7fb0580fcdc0 2024-04-16T05:40:17.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.251+0000 7fb054c00700 1 --2- 172.21.15.27:0/633923326 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb0400692b0 0x7fb04006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:17.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.251+0000 7fb054c00700 1 -- 172.21.15.27:0/633923326 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fb04c086cf0 con 0x7fb0580fcdc0 2024-04-16T05:40:17.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.251+0000 7fb057e00700 1 --2- 172.21.15.27:0/633923326 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb0400692b0 0x7fb04006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:17.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.252+0000 7fb057e00700 1 --2- 172.21.15.27:0/633923326 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb0400692b0 0x7fb04006b760 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb04400b0d0 tx=0x7fb044009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:17.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.258+0000 7fb054c00700 1 -- 172.21.15.27:0/633923326 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb04c056a30 con 0x7fb0580fcdc0 2024-04-16T05:40:17.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.530+0000 7fb05f400700 1 -- 172.21.15.27:0/633923326 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb05810f7a0 con 0x7fb0580fcdc0 2024-04-16T05:40:17.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.531+0000 7fb054c00700 1 -- 172.21.15.27:0/633923326 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 4 v4) v1 ==== 75+0+1247 (secure 0 0 0) 0x7fb04c0565c0 con 0x7fb0580fcdc0 2024-04-16T05:40:17.530 INFO:teuthology.orchestra.run.smithi027.stdout:e4 2024-04-16T05:40:17.530 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:40:17.530 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:40:17.530 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:40:17.530 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:40:17.530 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:40:17.530 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:40:17.531 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 4 2024-04-16T05:40:17.531 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:40:17.531 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:40:17.531 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:40:16.905522+0000 2024-04-16T05:40:17.531 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:40:17.531 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:40:17.531 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:40:17.531 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:40:17.531 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:40:17.531 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:40:17.531 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:40:17.531 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 0 2024-04-16T05:40:17.531 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:40:17.531 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:40:17.531 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:40:17.532 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=14446} 2024-04-16T05:40:17.532 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:40:17.532 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:40:17.532 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:40:17.532 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:40:17.532 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:40:17.532 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:40:17.532 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:40:17.532 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 0 2024-04-16T05:40:17.532 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{0:14446} state up:creating seq 1 addr [v2:172.21.15.27:6826/1980301393,v1:172.21.15.27:6827/1980301393] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:40:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:40:17.532 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:40:17.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.534+0000 7fb05f400700 1 -- 172.21.15.27:0/633923326 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb0400692b0 msgr2=0x7fb04006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:17.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.534+0000 7fb05f400700 1 --2- 172.21.15.27:0/633923326 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb0400692b0 0x7fb04006b760 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb04400b0d0 tx=0x7fb044009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:17.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.534+0000 7fb05f400700 1 -- 172.21.15.27:0/633923326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0580fcdc0 msgr2=0x7fb05810da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:17.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.534+0000 7fb05f400700 1 --2- 172.21.15.27:0/633923326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0580fcdc0 0x7fb05810da30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb04c00eb40 tx=0x7fb04c00b040 comp rx=0 tx=0).stop 2024-04-16T05:40:17.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.534+0000 7fb05f400700 1 -- 172.21.15.27:0/633923326 shutdown_connections 2024-04-16T05:40:17.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.534+0000 7fb05f400700 1 --2- 172.21.15.27:0/633923326 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb0400692b0 0x7fb04006b760 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:17.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.534+0000 7fb05f400700 1 --2- 172.21.15.27:0/633923326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0580fcdc0 0x7fb05810da30 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:17.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.534+0000 7fb05f400700 1 --2- 172.21.15.27:0/633923326 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb0580fd710 0x7fb05810df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:17.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.534+0000 7fb05f400700 1 -- 172.21.15.27:0/633923326 >> 172.21.15.27:0/633923326 conn(0x7fb0580f8540 msgr2=0x7fb0580f9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:17.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.535+0000 7fb05f400700 1 -- 172.21.15.27:0/633923326 shutdown_connections 2024-04-16T05:40:17.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:17.535+0000 7fb05f400700 1 -- 172.21.15.27:0/633923326 wait complete. 2024-04-16T05:40:17.534 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 4 2024-04-16T05:40:18.212 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:17 smithi122 ceph-mon[35998]: mds.? [v2:172.21.15.27:6826/1980301393,v1:172.21.15.27:6827/1980301393] up:boot 2024-04-16T05:40:18.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:17 smithi122 ceph-mon[35998]: daemon mds.cephfs.smithi027.hkhpuq assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-16T05:40:18.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:17 smithi122 ceph-mon[35998]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T05:40:18.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:17 smithi122 ceph-mon[35998]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-16T05:40:18.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:17 smithi122 ceph-mon[35998]: Cluster is now healthy 2024-04-16T05:40:18.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:17 smithi122 ceph-mon[35998]: fsmap cephfs:0 1 up:standby 2024-04-16T05:40:18.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:17 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.hkhpuq"}]: dispatch 2024-04-16T05:40:18.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:17 smithi122 ceph-mon[35998]: fsmap cephfs:1 {0=cephfs.smithi027.hkhpuq=up:creating} 2024-04-16T05:40:18.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:17 smithi122 ceph-mon[35998]: daemon mds.cephfs.smithi027.hkhpuq is now active in filesystem cephfs as rank 0 2024-04-16T05:40:18.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:17 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:18.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:17 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:18.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:17 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:18.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:17 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.lojjdp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:40:18.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:17 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.lojjdp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T05:40:18.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:17 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:40:18.213 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:17 smithi122 ceph-mon[35998]: Deploying daemon mds.cephfs.smithi122.lojjdp on smithi122 2024-04-16T05:40:18.214 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:17 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/633923326' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:40:18.214 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:17 smithi122 ceph-mon[35998]: pgmap v227: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:18.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:17 smithi027 ceph-mon[29206]: mds.? [v2:172.21.15.27:6826/1980301393,v1:172.21.15.27:6827/1980301393] up:boot 2024-04-16T05:40:18.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:17 smithi027 ceph-mon[29206]: daemon mds.cephfs.smithi027.hkhpuq assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-04-16T05:40:18.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:17 smithi027 ceph-mon[29206]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T05:40:18.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:17 smithi027 ceph-mon[29206]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-16T05:40:18.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:17 smithi027 ceph-mon[29206]: Cluster is now healthy 2024-04-16T05:40:18.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:17 smithi027 ceph-mon[29206]: fsmap cephfs:0 1 up:standby 2024-04-16T05:40:18.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:17 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.hkhpuq"}]: dispatch 2024-04-16T05:40:18.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:17 smithi027 ceph-mon[29206]: fsmap cephfs:1 {0=cephfs.smithi027.hkhpuq=up:creating} 2024-04-16T05:40:18.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:17 smithi027 ceph-mon[29206]: daemon mds.cephfs.smithi027.hkhpuq is now active in filesystem cephfs as rank 0 2024-04-16T05:40:18.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:17 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:18.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:17 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:18.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:17 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:18.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:17 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.lojjdp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:40:18.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:17 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.lojjdp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T05:40:18.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:17 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:40:18.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:17 smithi027 ceph-mon[29206]: Deploying daemon mds.cephfs.smithi122.lojjdp on smithi122 2024-04-16T05:40:18.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:17 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/633923326' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:40:18.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:17 smithi027 ceph-mon[29206]: pgmap v227: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:18.425 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T05:40:18.436 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-16T05:40:18.436 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs set cephfs max_mds 1' 2024-04-16T05:40:18.949 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:40:19.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:18 smithi122 ceph-mon[35998]: mds.? [v2:172.21.15.27:6826/1980301393,v1:172.21.15.27:6827/1980301393] up:active 2024-04-16T05:40:19.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:18 smithi122 ceph-mon[35998]: fsmap cephfs:1 {0=cephfs.smithi027.hkhpuq=up:active} 2024-04-16T05:40:19.275 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:18 smithi027 ceph-mon[29206]: mds.? [v2:172.21.15.27:6826/1980301393,v1:172.21.15.27:6827/1980301393] up:active 2024-04-16T05:40:19.275 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:18 smithi027 ceph-mon[29206]: fsmap cephfs:1 {0=cephfs.smithi027.hkhpuq=up:active} 2024-04-16T05:40:20.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:19 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:40:20.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:19 smithi122 ceph-mon[35998]: pgmap v228: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 530 B/s wr, 2 op/s 2024-04-16T05:40:20.291 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:19 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:40:20.291 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:19 smithi027 ceph-mon[29206]: pgmap v228: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 530 B/s wr, 2 op/s 2024-04-16T05:40:21.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.092+0000 7f2a57400700 1 -- 172.21.15.27:0/3927247402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a500feff0 msgr2=0x7f2a500ff400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:21.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.092+0000 7f2a57400700 1 --2- 172.21.15.27:0/3927247402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a500feff0 0x7f2a500ff400 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2a44004940 tx=0x7f2a44009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:21.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.092+0000 7f2a57400700 1 -- 172.21.15.27:0/3927247402 shutdown_connections 2024-04-16T05:40:21.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.092+0000 7f2a57400700 1 --2- 172.21.15.27:0/3927247402 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a501002a0 0x7f2a50100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:21.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.092+0000 7f2a57400700 1 --2- 172.21.15.27:0/3927247402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a500feff0 0x7f2a500ff400 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:21.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.092+0000 7f2a57400700 1 -- 172.21.15.27:0/3927247402 >> 172.21.15.27:0/3927247402 conn(0x7f2a500fa790 msgr2=0x7f2a500fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:21.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.092+0000 7f2a57400700 1 -- 172.21.15.27:0/3927247402 shutdown_connections 2024-04-16T05:40:21.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.092+0000 7f2a57400700 1 -- 172.21.15.27:0/3927247402 wait complete. 2024-04-16T05:40:21.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.093+0000 7f2a57400700 1 Processor -- start 2024-04-16T05:40:21.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.093+0000 7f2a57400700 1 -- start start 2024-04-16T05:40:21.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.094+0000 7f2a57400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a500feff0 0x7f2a5010fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:21.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.094+0000 7f2a57400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a501002a0 0x7f2a50110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:21.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.094+0000 7f2a57400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a50110680 con 0x7f2a501002a0 2024-04-16T05:40:21.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.094+0000 7f2a57400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a501107f0 con 0x7f2a500feff0 2024-04-16T05:40:21.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.094+0000 7f2a4fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a501002a0 0x7f2a50110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:21.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.094+0000 7f2a4fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a501002a0 0x7f2a50110080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35192/0 (socket says 172.21.15.27:35192) 2024-04-16T05:40:21.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.094+0000 7f2a4fe00700 1 -- 172.21.15.27:0/3232813067 learned_addr learned my addr 172.21.15.27:0/3232813067 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:40:21.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.094+0000 7f2a54a00700 1 --2- 172.21.15.27:0/3232813067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a500feff0 0x7f2a5010fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:21.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.095+0000 7f2a4fe00700 1 -- 172.21.15.27:0/3232813067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a500feff0 msgr2=0x7f2a5010fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:21.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.095+0000 7f2a4fe00700 1 --2- 172.21.15.27:0/3232813067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a500feff0 0x7f2a5010fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:21.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.095+0000 7f2a4fe00700 1 -- 172.21.15.27:0/3232813067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a44005040 con 0x7f2a501002a0 2024-04-16T05:40:21.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.095+0000 7f2a4fe00700 1 --2- 172.21.15.27:0/3232813067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a501002a0 0x7f2a50110080 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c00c830 tx=0x7f2a3c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:21.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.095+0000 7f2a4cc00700 1 -- 172.21.15.27:0/3232813067 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a3c00a760 con 0x7f2a501002a0 2024-04-16T05:40:21.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.095+0000 7f2a4cc00700 1 -- 172.21.15.27:0/3232813067 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a3c00ad90 con 0x7f2a501002a0 2024-04-16T05:40:21.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.095+0000 7f2a57400700 1 -- 172.21.15.27:0/3232813067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a50115250 con 0x7f2a501002a0 2024-04-16T05:40:21.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.095+0000 7f2a4cc00700 1 -- 172.21.15.27:0/3232813067 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a3c0113a0 con 0x7f2a501002a0 2024-04-16T05:40:21.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.096+0000 7f2a57400700 1 -- 172.21.15.27:0/3232813067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a50115810 con 0x7f2a501002a0 2024-04-16T05:40:21.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.097+0000 7f2a57400700 1 -- 172.21.15.27:0/3232813067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a5004c710 con 0x7f2a501002a0 2024-04-16T05:40:21.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.099+0000 7f2a4cc00700 1 -- 172.21.15.27:0/3232813067 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f2a3c012400 con 0x7f2a501002a0 2024-04-16T05:40:21.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.100+0000 7f2a4cc00700 1 --2- 172.21.15.27:0/3232813067 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f2a380692a0 0x7f2a3806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:21.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.100+0000 7f2a4cc00700 1 -- 172.21.15.27:0/3232813067 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f2a3c087c40 con 0x7f2a501002a0 2024-04-16T05:40:21.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.100+0000 7f2a54a00700 1 --2- 172.21.15.27:0/3232813067 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f2a380692a0 0x7f2a3806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:21.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.101+0000 7f2a54a00700 1 --2- 172.21.15.27:0/3232813067 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f2a380692a0 0x7f2a3806b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2a44004f90 tx=0x7f2a44009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:21.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.103+0000 7f2a4cc00700 1 -- 172.21.15.27:0/3232813067 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f2a3c057aa0 con 0x7f2a501002a0 2024-04-16T05:40:21.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.380+0000 7f2a57400700 1 -- 172.21.15.27:0/3232813067 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f2a50107c70 con 0x7f2a501002a0 2024-04-16T05:40:21.691 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:21.691 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:21.691 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:21.691 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.ejoytk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:40:21.691 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.ejoytk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T05:40:21.692 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:40:21.692 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:21 smithi027 ceph-mon[29206]: Deploying daemon mds.cephfs.smithi027.ejoytk on smithi027 2024-04-16T05:40:21.692 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:21 smithi027 ceph-mon[29206]: mds.? [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] up:boot 2024-04-16T05:40:21.692 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:21 smithi027 ceph-mon[29206]: fsmap cephfs:1 {0=cephfs.smithi027.hkhpuq=up:active} 1 up:standby 2024-04-16T05:40:21.692 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:21 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.lojjdp"}]: dispatch 2024-04-16T05:40:21.692 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:21 smithi027 ceph-mon[29206]: fsmap cephfs:1 {0=cephfs.smithi027.hkhpuq=up:active} 1 up:standby 2024-04-16T05:40:21.692 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:21 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/3232813067' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-04-16T05:40:21.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.947+0000 7f2a4cc00700 1 -- 172.21.15.27:0/3232813067 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v8) v1 ==== 105+0+0 (secure 0 0 0) 0x7f2a50107c70 con 0x7f2a501002a0 2024-04-16T05:40:21.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.951+0000 7f2a57400700 1 -- 172.21.15.27:0/3232813067 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f2a380692a0 msgr2=0x7f2a3806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:21.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.951+0000 7f2a57400700 1 --2- 172.21.15.27:0/3232813067 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f2a380692a0 0x7f2a3806b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2a44004f90 tx=0x7f2a44009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:21.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.951+0000 7f2a57400700 1 -- 172.21.15.27:0/3232813067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a501002a0 msgr2=0x7f2a50110080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:21.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.951+0000 7f2a57400700 1 --2- 172.21.15.27:0/3232813067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a501002a0 0x7f2a50110080 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c00c830 tx=0x7f2a3c004470 comp rx=0 tx=0).stop 2024-04-16T05:40:21.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.952+0000 7f2a57400700 1 -- 172.21.15.27:0/3232813067 shutdown_connections 2024-04-16T05:40:21.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.952+0000 7f2a57400700 1 --2- 172.21.15.27:0/3232813067 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f2a380692a0 0x7f2a3806b750 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:21.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.952+0000 7f2a57400700 1 --2- 172.21.15.27:0/3232813067 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a500feff0 0x7f2a5010fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:21.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.952+0000 7f2a57400700 1 --2- 172.21.15.27:0/3232813067 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a501002a0 0x7f2a50110080 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:21.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.952+0000 7f2a57400700 1 -- 172.21.15.27:0/3232813067 >> 172.21.15.27:0/3232813067 conn(0x7f2a500fa790 msgr2=0x7f2a500fbe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:21.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.952+0000 7f2a57400700 1 -- 172.21.15.27:0/3232813067 shutdown_connections 2024-04-16T05:40:21.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:21.952+0000 7f2a57400700 1 -- 172.21.15.27:0/3232813067 wait complete. 2024-04-16T05:40:22.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:22.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:22.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:22.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.ejoytk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:40:22.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.ejoytk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T05:40:22.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:40:22.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:21 smithi122 ceph-mon[35998]: Deploying daemon mds.cephfs.smithi027.ejoytk on smithi027 2024-04-16T05:40:22.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:21 smithi122 ceph-mon[35998]: mds.? [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] up:boot 2024-04-16T05:40:22.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:21 smithi122 ceph-mon[35998]: fsmap cephfs:1 {0=cephfs.smithi027.hkhpuq=up:active} 1 up:standby 2024-04-16T05:40:22.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:21 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.lojjdp"}]: dispatch 2024-04-16T05:40:22.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:21 smithi122 ceph-mon[35998]: fsmap cephfs:1 {0=cephfs.smithi027.hkhpuq=up:active} 1 up:standby 2024-04-16T05:40:22.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:21 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/3232813067' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-04-16T05:40:22.807 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T05:40:22.818 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-16T05:40:22.818 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-04-16T05:40:22.853 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:22 smithi027 ceph-mon[29206]: pgmap v229: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 2 op/s 2024-04-16T05:40:22.853 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:22 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/3232813067' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-04-16T05:40:22.853 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:22 smithi027 ceph-mon[29206]: fsmap cephfs:1 {0=cephfs.smithi027.hkhpuq=up:active} 1 up:standby 2024-04-16T05:40:23.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:22 smithi122 ceph-mon[35998]: pgmap v229: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 2 op/s 2024-04-16T05:40:23.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:22 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/3232813067' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-04-16T05:40:23.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:22 smithi122 ceph-mon[35998]: fsmap cephfs:1 {0=cephfs.smithi027.hkhpuq=up:active} 1 up:standby 2024-04-16T05:40:23.380 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:40:24.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:24 smithi027 ceph-mon[29206]: pgmap v230: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-16T05:40:25.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:24 smithi122 ceph-mon[35998]: pgmap v230: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-16T05:40:25.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.851+0000 7f8b87600700 1 -- 172.21.15.27:0/2519207277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b80070890 msgr2=0x7f8b80070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:25.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.851+0000 7f8b87600700 1 --2- 172.21.15.27:0/2519207277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b80070890 0x7f8b80070d00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c004940 tx=0x7f8b6c009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:25.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.852+0000 7f8b87600700 1 -- 172.21.15.27:0/2519207277 shutdown_connections 2024-04-16T05:40:25.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.852+0000 7f8b87600700 1 --2- 172.21.15.27:0/2519207277 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b80070890 0x7f8b80070d00 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:25.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.852+0000 7f8b87600700 1 --2- 172.21.15.27:0/2519207277 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8b80071e40 0x7f8b80070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:25.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.852+0000 7f8b87600700 1 -- 172.21.15.27:0/2519207277 >> 172.21.15.27:0/2519207277 conn(0x7f8b800f8540 msgr2=0x7f8b800fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:25.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.852+0000 7f8b87600700 1 -- 172.21.15.27:0/2519207277 shutdown_connections 2024-04-16T05:40:25.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.852+0000 7f8b87600700 1 -- 172.21.15.27:0/2519207277 wait complete. 2024-04-16T05:40:25.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.853+0000 7f8b87600700 1 Processor -- start 2024-04-16T05:40:25.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.853+0000 7f8b87600700 1 -- start start 2024-04-16T05:40:25.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.854+0000 7f8b87600700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8b80070890 0x7f8b8010faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:25.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.854+0000 7f8b87600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b80071e40 0x7f8b80110030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:25.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.854+0000 7f8b87600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b80110630 con 0x7f8b80071e40 2024-04-16T05:40:25.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.854+0000 7f8b87600700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b801107a0 con 0x7f8b80070890 2024-04-16T05:40:25.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.854+0000 7f8b7fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b80071e40 0x7f8b80110030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:25.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.854+0000 7f8b7fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b80071e40 0x7f8b80110030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51094/0 (socket says 172.21.15.27:51094) 2024-04-16T05:40:25.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.854+0000 7f8b7fe00700 1 -- 172.21.15.27:0/3978387440 learned_addr learned my addr 172.21.15.27:0/3978387440 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:40:25.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.854+0000 7f8b84c00700 1 --2- 172.21.15.27:0/3978387440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8b80070890 0x7f8b8010faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:25.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.855+0000 7f8b7fe00700 1 -- 172.21.15.27:0/3978387440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8b80070890 msgr2=0x7f8b8010faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:25.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.855+0000 7f8b7fe00700 1 --2- 172.21.15.27:0/3978387440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8b80070890 0x7f8b8010faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:25.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.855+0000 7f8b7fe00700 1 -- 172.21.15.27:0/3978387440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6c005040 con 0x7f8b80071e40 2024-04-16T05:40:25.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.855+0000 7f8b7fe00700 1 --2- 172.21.15.27:0/3978387440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b80071e40 0x7f8b80110030 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c00b0d0 tx=0x7f8b6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:25.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.855+0000 7f8b7cc00700 1 -- 172.21.15.27:0/3978387440 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b6c00e030 con 0x7f8b80071e40 2024-04-16T05:40:25.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.855+0000 7f8b87600700 1 -- 172.21.15.27:0/3978387440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b801151a0 con 0x7f8b80071e40 2024-04-16T05:40:25.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.855+0000 7f8b7cc00700 1 -- 172.21.15.27:0/3978387440 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b6c01c400 con 0x7f8b80071e40 2024-04-16T05:40:25.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.855+0000 7f8b7cc00700 1 -- 172.21.15.27:0/3978387440 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8b6c00b630 con 0x7f8b80071e40 2024-04-16T05:40:25.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.855+0000 7f8b87600700 1 -- 172.21.15.27:0/3978387440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b80115760 con 0x7f8b80071e40 2024-04-16T05:40:25.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.857+0000 7f8b87600700 1 -- 172.21.15.27:0/3978387440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b8004c710 con 0x7f8b80071e40 2024-04-16T05:40:25.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.858+0000 7f8b7cc00700 1 -- 172.21.15.27:0/3978387440 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f8b6c013af0 con 0x7f8b80071e40 2024-04-16T05:40:25.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.859+0000 7f8b7cc00700 1 --2- 172.21.15.27:0/3978387440 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f8b680692a0 0x7f8b6806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:25.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.859+0000 7f8b7cc00700 1 -- 172.21.15.27:0/3978387440 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f8b6c0889c0 con 0x7f8b80071e40 2024-04-16T05:40:25.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.859+0000 7f8b84c00700 1 --2- 172.21.15.27:0/3978387440 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f8b680692a0 0x7f8b6806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:25.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.860+0000 7f8b84c00700 1 --2- 172.21.15.27:0/3978387440 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f8b680692a0 0x7f8b6806b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009b80 tx=0x7f8b7400b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:25.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:25.866+0000 7f8b7cc00700 1 -- 172.21.15.27:0/3978387440 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f8b6c0586e0 con 0x7f8b80071e40 2024-04-16T05:40:26.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:25 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:26.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:25 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:26.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:26.143+0000 7f8b87600700 1 -- 172.21.15.27:0/3978387440 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f8b80107b00 con 0x7f8b80071e40 2024-04-16T05:40:26.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:26.767+0000 7f8b7cc00700 1 -- 172.21.15.27:0/3978387440 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8b6c004490 con 0x7f8b80071e40 2024-04-16T05:40:26.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:26.771+0000 7f8b87600700 1 -- 172.21.15.27:0/3978387440 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f8b680692a0 msgr2=0x7f8b6806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:26.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:26.771+0000 7f8b87600700 1 --2- 172.21.15.27:0/3978387440 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f8b680692a0 0x7f8b6806b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009b80 tx=0x7f8b7400b040 comp rx=0 tx=0).stop 2024-04-16T05:40:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:26.771+0000 7f8b87600700 1 -- 172.21.15.27:0/3978387440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b80071e40 msgr2=0x7f8b80110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:26.771+0000 7f8b87600700 1 --2- 172.21.15.27:0/3978387440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b80071e40 0x7f8b80110030 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c00b0d0 tx=0x7f8b6c009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:26.771+0000 7f8b87600700 1 -- 172.21.15.27:0/3978387440 shutdown_connections 2024-04-16T05:40:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:26.771+0000 7f8b87600700 1 --2- 172.21.15.27:0/3978387440 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f8b680692a0 0x7f8b6806b750 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:26.771+0000 7f8b87600700 1 --2- 172.21.15.27:0/3978387440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8b80070890 0x7f8b8010faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:26.771+0000 7f8b87600700 1 --2- 172.21.15.27:0/3978387440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8b80071e40 0x7f8b80110030 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:26.771+0000 7f8b87600700 1 -- 172.21.15.27:0/3978387440 >> 172.21.15.27:0/3978387440 conn(0x7f8b800f8540 msgr2=0x7f8b800f9b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:26.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:26.772+0000 7f8b87600700 1 -- 172.21.15.27:0/3978387440 shutdown_connections 2024-04-16T05:40:26.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:26.772+0000 7f8b87600700 1 -- 172.21.15.27:0/3978387440 wait complete. 2024-04-16T05:40:26.866 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:26 smithi122 ceph-mon[35998]: pgmap v231: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-16T05:40:26.866 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:26 smithi122 ceph-mon[35998]: mds.? [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] up:boot 2024-04-16T05:40:26.866 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:26 smithi122 ceph-mon[35998]: fsmap cephfs:1 {0=cephfs.smithi027.hkhpuq=up:active} 2 up:standby 2024-04-16T05:40:26.866 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:26 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.ejoytk"}]: dispatch 2024-04-16T05:40:26.866 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:26 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:26.866 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:26 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:26.866 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:26 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:26.867 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:26 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.rulewi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:40:26.867 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:26 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.rulewi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T05:40:26.867 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:26 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:40:26.867 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:26 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/3978387440' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-04-16T05:40:27.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:26 smithi027 ceph-mon[29206]: pgmap v231: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-16T05:40:27.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:26 smithi027 ceph-mon[29206]: mds.? [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] up:boot 2024-04-16T05:40:27.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:26 smithi027 ceph-mon[29206]: fsmap cephfs:1 {0=cephfs.smithi027.hkhpuq=up:active} 2 up:standby 2024-04-16T05:40:27.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:26 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.ejoytk"}]: dispatch 2024-04-16T05:40:27.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:26 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:27.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:26 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:27.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:26 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:27.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:26 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.rulewi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:40:27.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:26 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.rulewi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-16T05:40:27.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:26 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:40:27.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:26 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/3978387440' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-04-16T05:40:27.697 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T05:40:27.709 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-16T05:40:27.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs set cephfs inline_data false' 2024-04-16T05:40:28.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:27 smithi122 ceph-mon[35998]: Deploying daemon mds.cephfs.smithi122.rulewi on smithi122 2024-04-16T05:40:28.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:27 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/3978387440' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-04-16T05:40:28.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:27 smithi122 ceph-mon[35998]: fsmap cephfs:1 {0=cephfs.smithi027.hkhpuq=up:active} 2 up:standby 2024-04-16T05:40:28.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:27 smithi027 ceph-mon[29206]: Deploying daemon mds.cephfs.smithi122.rulewi on smithi122 2024-04-16T05:40:28.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:27 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/3978387440' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-04-16T05:40:28.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:27 smithi027 ceph-mon[29206]: fsmap cephfs:1 {0=cephfs.smithi027.hkhpuq=up:active} 2 up:standby 2024-04-16T05:40:28.262 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:40:28.873 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:28 smithi122 ceph-mon[35998]: pgmap v232: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-16T05:40:29.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:28 smithi027 ceph-mon[29206]: pgmap v232: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-16T05:40:30.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.606+0000 7f1430600700 1 -- 172.21.15.27:0/1244123333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14280fce80 msgr2=0x7f14280fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:30.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.606+0000 7f1430600700 1 --2- 172.21.15.27:0/1244123333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14280fce80 0x7f14280fd290 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f141c004940 tx=0x7f141c009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:30.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.606+0000 7f1430600700 1 -- 172.21.15.27:0/1244123333 shutdown_connections 2024-04-16T05:40:30.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.606+0000 7f1430600700 1 --2- 172.21.15.27:0/1244123333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14280fdff0 0x7f14280fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:30.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.606+0000 7f1430600700 1 --2- 172.21.15.27:0/1244123333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14280fce80 0x7f14280fd290 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:30.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.606+0000 7f1430600700 1 -- 172.21.15.27:0/1244123333 >> 172.21.15.27:0/1244123333 conn(0x7f14280f8540 msgr2=0x7f14280fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:30.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.606+0000 7f1430600700 1 -- 172.21.15.27:0/1244123333 shutdown_connections 2024-04-16T05:40:30.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.606+0000 7f1430600700 1 -- 172.21.15.27:0/1244123333 wait complete. 2024-04-16T05:40:30.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.607+0000 7f1430600700 1 Processor -- start 2024-04-16T05:40:30.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.607+0000 7f1430600700 1 -- start start 2024-04-16T05:40:30.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.607+0000 7f1430600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14280fce80 0x7f142810fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:30.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.607+0000 7f1430600700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14280fdff0 0x7f14281100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:30.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.608+0000 7f1430600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14281106b0 con 0x7f14280fce80 2024-04-16T05:40:30.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.608+0000 7f1430600700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1428110820 con 0x7f14280fdff0 2024-04-16T05:40:30.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.608+0000 7f142dc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14280fce80 0x7f142810fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:30.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.608+0000 7f142dc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14280fce80 0x7f142810fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:51128/0 (socket says 172.21.15.27:51128) 2024-04-16T05:40:30.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.608+0000 7f142dc00700 1 -- 172.21.15.27:0/1224420168 learned_addr learned my addr 172.21.15.27:0/1224420168 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:40:30.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.608+0000 7f142d200700 1 --2- 172.21.15.27:0/1224420168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14280fdff0 0x7f14281100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:30.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.608+0000 7f142dc00700 1 -- 172.21.15.27:0/1224420168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14280fdff0 msgr2=0x7f14281100b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:30.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.608+0000 7f142dc00700 1 --2- 172.21.15.27:0/1224420168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14280fdff0 0x7f14281100b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:30.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.608+0000 7f142dc00700 1 -- 172.21.15.27:0/1224420168 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f141c005040 con 0x7f14280fce80 2024-04-16T05:40:30.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.608+0000 7f142d200700 1 --2- 172.21.15.27:0/1224420168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14280fdff0 0x7f14281100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T05:40:30.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.609+0000 7f142dc00700 1 --2- 172.21.15.27:0/1224420168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14280fce80 0x7f142810fb70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f141c009150 tx=0x7f141c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:30.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.609+0000 7f1425600700 1 -- 172.21.15.27:0/1224420168 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f141c00e040 con 0x7f14280fce80 2024-04-16T05:40:30.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.609+0000 7f1425600700 1 -- 172.21.15.27:0/1224420168 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f141c00bbf0 con 0x7f14280fce80 2024-04-16T05:40:30.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.609+0000 7f1430600700 1 -- 172.21.15.27:0/1224420168 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1428115220 con 0x7f14280fce80 2024-04-16T05:40:30.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.609+0000 7f1425600700 1 -- 172.21.15.27:0/1224420168 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f141c01c630 con 0x7f14280fce80 2024-04-16T05:40:30.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.609+0000 7f1430600700 1 -- 172.21.15.27:0/1224420168 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14281157e0 con 0x7f14280fce80 2024-04-16T05:40:30.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.611+0000 7f1430600700 1 -- 172.21.15.27:0/1224420168 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1428101980 con 0x7f14280fce80 2024-04-16T05:40:30.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.612+0000 7f1425600700 1 -- 172.21.15.27:0/1224420168 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f141c00bd60 con 0x7f14280fce80 2024-04-16T05:40:30.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.613+0000 7f1425600700 1 --2- 172.21.15.27:0/1224420168 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f14100692a0 0x7f141006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:30.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.613+0000 7f1425600700 1 -- 172.21.15.27:0/1224420168 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f141c088610 con 0x7f14280fce80 2024-04-16T05:40:30.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.613+0000 7f142d200700 1 --2- 172.21.15.27:0/1224420168 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f14100692a0 0x7f141006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:30.613 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:30 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:30.613 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:30 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:30.614 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:30 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:30.614 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:30 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:30.614 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:30 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:30.614 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:30 smithi027 ceph-mon[29206]: pgmap v233: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-16T05:40:30.614 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:30 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:30.614 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:30 smithi027 ceph-mon[29206]: mds.? [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] up:boot 2024-04-16T05:40:30.614 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:30 smithi027 ceph-mon[29206]: mds.? [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] up:standby 2024-04-16T05:40:30.614 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:30 smithi027 ceph-mon[29206]: Dropping low affinity active daemon mds.cephfs.smithi027.hkhpuq in favor of higher affinity standby. 2024-04-16T05:40:30.614 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:30 smithi027 ceph-mon[29206]: Replacing daemon mds.cephfs.smithi027.hkhpuq as rank 0 with standby daemon mds.cephfs.smithi027.ejoytk 2024-04-16T05:40:30.614 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:30 smithi027 ceph-mon[29206]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T05:40:30.614 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:30 smithi027 ceph-mon[29206]: fsmap cephfs:1 {0=cephfs.smithi027.hkhpuq=up:active} 3 up:standby 2024-04-16T05:40:30.615 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:30 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.rulewi"}]: dispatch 2024-04-16T05:40:30.615 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:30 smithi027 ceph-mon[29206]: osdmap e46: 6 total, 6 up, 6 in 2024-04-16T05:40:30.615 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:30 smithi027 ceph-mon[29206]: fsmap cephfs:1/1 {0=cephfs.smithi027.ejoytk=up:replay} 2 up:standby 2024-04-16T05:40:30.615 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:30 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:40:30.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.614+0000 7f142d200700 1 --2- 172.21.15.27:0/1224420168 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f14100692a0 0x7f141006b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1414009fd0 tx=0x7f1414009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:30.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.618+0000 7f1425600700 1 -- 172.21.15.27:0/1224420168 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f141c058410 con 0x7f14280fce80 2024-04-16T05:40:30.735 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:30 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:30.735 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:30 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:30.736 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:30 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:30.736 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:30 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:30.736 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:30 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:30.736 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:30 smithi122 ceph-mon[35998]: pgmap v233: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-16T05:40:30.736 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:30 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:30.736 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:30 smithi122 ceph-mon[35998]: mds.? [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] up:boot 2024-04-16T05:40:30.736 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:30 smithi122 ceph-mon[35998]: mds.? [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] up:standby 2024-04-16T05:40:30.736 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:30 smithi122 ceph-mon[35998]: Dropping low affinity active daemon mds.cephfs.smithi027.hkhpuq in favor of higher affinity standby. 2024-04-16T05:40:30.736 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:30 smithi122 ceph-mon[35998]: Replacing daemon mds.cephfs.smithi027.hkhpuq as rank 0 with standby daemon mds.cephfs.smithi027.ejoytk 2024-04-16T05:40:30.736 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:30 smithi122 ceph-mon[35998]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T05:40:30.737 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:30 smithi122 ceph-mon[35998]: fsmap cephfs:1 {0=cephfs.smithi027.hkhpuq=up:active} 3 up:standby 2024-04-16T05:40:30.737 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:30 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.rulewi"}]: dispatch 2024-04-16T05:40:30.737 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:30 smithi122 ceph-mon[35998]: osdmap e46: 6 total, 6 up, 6 in 2024-04-16T05:40:30.737 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:30 smithi122 ceph-mon[35998]: fsmap cephfs:1/1 {0=cephfs.smithi027.ejoytk=up:replay} 2 up:standby 2024-04-16T05:40:30.737 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:30 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:40:30.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:30.906+0000 7f1430600700 1 -- 172.21.15.27:0/1224420168 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"} v 0) v1 -- 0x7f14281058e0 con 0x7f14280fce80 2024-04-16T05:40:31.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:31.703+0000 7f1425600700 1 -- 172.21.15.27:0/1224420168 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]=0 inline data disabled v14) v1 ==== 133+0+0 (secure 0 0 0) 0x7f141c057fa0 con 0x7f14280fce80 2024-04-16T05:40:31.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:31.706+0000 7f1430600700 1 -- 172.21.15.27:0/1224420168 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f14100692a0 msgr2=0x7f141006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:31.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:31.706+0000 7f1430600700 1 --2- 172.21.15.27:0/1224420168 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f14100692a0 0x7f141006b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1414009fd0 tx=0x7f1414009ee0 comp rx=0 tx=0).stop 2024-04-16T05:40:31.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:31.706+0000 7f1430600700 1 -- 172.21.15.27:0/1224420168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14280fce80 msgr2=0x7f142810fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:31.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:31.706+0000 7f1430600700 1 --2- 172.21.15.27:0/1224420168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14280fce80 0x7f142810fb70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f141c009150 tx=0x7f141c009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:31.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:31.706+0000 7f1430600700 1 -- 172.21.15.27:0/1224420168 shutdown_connections 2024-04-16T05:40:31.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:31.706+0000 7f1430600700 1 --2- 172.21.15.27:0/1224420168 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f14100692a0 0x7f141006b750 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:31.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:31.706+0000 7f1430600700 1 --2- 172.21.15.27:0/1224420168 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14280fce80 0x7f142810fb70 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:31.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:31.706+0000 7f1430600700 1 --2- 172.21.15.27:0/1224420168 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14280fdff0 0x7f14281100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:31.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:31.706+0000 7f1430600700 1 -- 172.21.15.27:0/1224420168 >> 172.21.15.27:0/1224420168 conn(0x7f14280f8540 msgr2=0x7f14280f9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:31.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:31.706+0000 7f1430600700 1 -- 172.21.15.27:0/1224420168 shutdown_connections 2024-04-16T05:40:31.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:31.707+0000 7f1430600700 1 -- 172.21.15.27:0/1224420168 wait complete. 2024-04-16T05:40:31.707 INFO:teuthology.orchestra.run.smithi027.stderr:inline data disabled 2024-04-16T05:40:31.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:31 smithi122 ceph-mon[35998]: mds.? [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] up:reconnect 2024-04-16T05:40:31.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:31 smithi122 ceph-mon[35998]: mds.? [v2:172.21.15.27:6826/2065690722,v1:172.21.15.27:6827/2065690722] up:boot 2024-04-16T05:40:31.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:31 smithi122 ceph-mon[35998]: fsmap cephfs:1/1 {0=cephfs.smithi027.ejoytk=up:reconnect} 3 up:standby 2024-04-16T05:40:31.899 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:31 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.hkhpuq"}]: dispatch 2024-04-16T05:40:31.900 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:31 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/1224420168' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-04-16T05:40:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:31 smithi027 ceph-mon[29206]: mds.? [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] up:reconnect 2024-04-16T05:40:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:31 smithi027 ceph-mon[29206]: mds.? [v2:172.21.15.27:6826/2065690722,v1:172.21.15.27:6827/2065690722] up:boot 2024-04-16T05:40:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:31 smithi027 ceph-mon[29206]: fsmap cephfs:1/1 {0=cephfs.smithi027.ejoytk=up:reconnect} 3 up:standby 2024-04-16T05:40:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:31 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.hkhpuq"}]: dispatch 2024-04-16T05:40:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:31 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1224420168' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]: dispatch 2024-04-16T05:40:32.586 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T05:40:32.596 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-16T05:40:32.596 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-16T05:40:32.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:32 smithi027 ceph-mon[29206]: pgmap v235: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 204 B/s rd, 204 B/s wr, 2 op/s 2024-04-16T05:40:32.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:32 smithi027 ceph-mon[29206]: mds.? [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] up:rejoin 2024-04-16T05:40:32.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:32 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1224420168' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-04-16T05:40:32.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:32 smithi027 ceph-mon[29206]: fsmap cephfs:1/1 {0=cephfs.smithi027.ejoytk=up:rejoin} 3 up:standby 2024-04-16T05:40:32.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:32 smithi027 ceph-mon[29206]: daemon mds.cephfs.smithi027.ejoytk is now active in filesystem cephfs as rank 0 2024-04-16T05:40:33.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:32 smithi122 ceph-mon[35998]: pgmap v235: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 204 B/s rd, 204 B/s wr, 2 op/s 2024-04-16T05:40:33.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:32 smithi122 ceph-mon[35998]: mds.? [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] up:rejoin 2024-04-16T05:40:33.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:32 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/1224420168' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "false"}]': finished 2024-04-16T05:40:33.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:32 smithi122 ceph-mon[35998]: fsmap cephfs:1/1 {0=cephfs.smithi027.ejoytk=up:rejoin} 3 up:standby 2024-04-16T05:40:33.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:32 smithi122 ceph-mon[35998]: daemon mds.cephfs.smithi027.ejoytk is now active in filesystem cephfs as rank 0 2024-04-16T05:40:33.515 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:40:34.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:33 smithi122 ceph-mon[35998]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T05:40:34.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:33 smithi122 ceph-mon[35998]: Cluster is now healthy 2024-04-16T05:40:34.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:33 smithi122 ceph-mon[35998]: mds.? [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] up:active 2024-04-16T05:40:34.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:33 smithi122 ceph-mon[35998]: mds.? [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] up:standby 2024-04-16T05:40:34.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:33 smithi122 ceph-mon[35998]: fsmap cephfs:1 {0=cephfs.smithi027.ejoytk=up:active} 3 up:standby 2024-04-16T05:40:34.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:33 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:34.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:33 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:34.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:33 smithi027 ceph-mon[29206]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T05:40:34.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:33 smithi027 ceph-mon[29206]: Cluster is now healthy 2024-04-16T05:40:34.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:33 smithi027 ceph-mon[29206]: mds.? [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] up:active 2024-04-16T05:40:34.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:33 smithi027 ceph-mon[29206]: mds.? [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] up:standby 2024-04-16T05:40:34.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:33 smithi027 ceph-mon[29206]: fsmap cephfs:1 {0=cephfs.smithi027.ejoytk=up:active} 3 up:standby 2024-04-16T05:40:34.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:33 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:34.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:33 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:35.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:34 smithi122 ceph-mon[35998]: pgmap v236: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 716 B/s rd, 0 B/s wr, 0 op/s 2024-04-16T05:40:35.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:34 smithi122 ceph-mon[35998]: mds.? [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] up:standby 2024-04-16T05:40:35.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:34 smithi122 ceph-mon[35998]: fsmap cephfs:1 {0=cephfs.smithi027.ejoytk=up:active} 3 up:standby 2024-04-16T05:40:35.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:34 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:40:35.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:34 smithi027 ceph-mon[29206]: pgmap v236: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 716 B/s rd, 0 B/s wr, 0 op/s 2024-04-16T05:40:35.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:34 smithi027 ceph-mon[29206]: mds.? [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] up:standby 2024-04-16T05:40:35.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:34 smithi027 ceph-mon[29206]: fsmap cephfs:1 {0=cephfs.smithi027.ejoytk=up:active} 3 up:standby 2024-04-16T05:40:35.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:34 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:40:36.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.179+0000 7fe291a00700 1 -- 172.21.15.27:0/2773369604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe28c066ce0 msgr2=0x7fe28c0ff820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:36.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.179+0000 7fe291a00700 1 --2- 172.21.15.27:0/2773369604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe28c066ce0 0x7fe28c0ff820 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe27c004940 tx=0x7fe27c009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:36.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.179+0000 7fe291a00700 1 -- 172.21.15.27:0/2773369604 shutdown_connections 2024-04-16T05:40:36.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.179+0000 7fe291a00700 1 --2- 172.21.15.27:0/2773369604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe28c0ffd60 0x7fe28c102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:36.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.179+0000 7fe291a00700 1 --2- 172.21.15.27:0/2773369604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe28c066ce0 0x7fe28c0ff820 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:36.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.179+0000 7fe291a00700 1 -- 172.21.15.27:0/2773369604 >> 172.21.15.27:0/2773369604 conn(0x7fe28c0f7290 msgr2=0x7fe28c0f96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:36.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.179+0000 7fe291a00700 1 -- 172.21.15.27:0/2773369604 shutdown_connections 2024-04-16T05:40:36.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.179+0000 7fe291a00700 1 -- 172.21.15.27:0/2773369604 wait complete. 2024-04-16T05:40:36.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.180+0000 7fe291a00700 1 Processor -- start 2024-04-16T05:40:36.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.180+0000 7fe291a00700 1 -- start start 2024-04-16T05:40:36.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.181+0000 7fe291a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe28c066ce0 0x7fe28c10d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:36.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.181+0000 7fe291a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe28c0ffd60 0x7fe28c10de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:36.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.181+0000 7fe291a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c10e400 con 0x7fe28c0ffd60 2024-04-16T05:40:36.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.181+0000 7fe291a00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c10ec00 con 0x7fe28c066ce0 2024-04-16T05:40:36.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.181+0000 7fe28b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe28c0ffd60 0x7fe28c10de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:36.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.181+0000 7fe28b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe28c0ffd60 0x7fe28c10de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45432/0 (socket says 172.21.15.27:45432) 2024-04-16T05:40:36.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.181+0000 7fe28b400700 1 -- 172.21.15.27:0/3618769946 learned_addr learned my addr 172.21.15.27:0/3618769946 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:40:36.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.181+0000 7fe28be00700 1 --2- 172.21.15.27:0/3618769946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe28c066ce0 0x7fe28c10d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:36.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.182+0000 7fe28b400700 1 -- 172.21.15.27:0/3618769946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe28c066ce0 msgr2=0x7fe28c10d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:36.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.182+0000 7fe28b400700 1 --2- 172.21.15.27:0/3618769946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe28c066ce0 0x7fe28c10d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:36.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.182+0000 7fe28b400700 1 -- 172.21.15.27:0/3618769946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe27c005040 con 0x7fe28c0ffd60 2024-04-16T05:40:36.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.182+0000 7fe28b400700 1 --2- 172.21.15.27:0/3618769946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe28c0ffd60 0x7fe28c10de00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe27800c810 tx=0x7fe278004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:36.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.183+0000 7fe283e00700 1 -- 172.21.15.27:0/3618769946 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe278012810 con 0x7fe28c0ffd60 2024-04-16T05:40:36.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.183+0000 7fe283e00700 1 -- 172.21.15.27:0/3618769946 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe278012e40 con 0x7fe28c0ffd60 2024-04-16T05:40:36.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.183+0000 7fe291a00700 1 -- 172.21.15.27:0/3618769946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28c10eee0 con 0x7fe28c0ffd60 2024-04-16T05:40:36.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.183+0000 7fe283e00700 1 -- 172.21.15.27:0/3618769946 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2780102e0 con 0x7fe28c0ffd60 2024-04-16T05:40:36.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.183+0000 7fe291a00700 1 -- 172.21.15.27:0/3618769946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28c10f420 con 0x7fe28c0ffd60 2024-04-16T05:40:36.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.184+0000 7fe291a00700 1 -- 172.21.15.27:0/3618769946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe28c0faf30 con 0x7fe28c0ffd60 2024-04-16T05:40:36.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.185+0000 7fe283e00700 1 -- 172.21.15.27:0/3618769946 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fe278011380 con 0x7fe28c0ffd60 2024-04-16T05:40:36.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.187+0000 7fe283e00700 1 --2- 172.21.15.27:0/3618769946 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fe2700692a0 0x7fe27006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:36.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.187+0000 7fe283e00700 1 -- 172.21.15.27:0/3618769946 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe278087690 con 0x7fe28c0ffd60 2024-04-16T05:40:36.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.187+0000 7fe28be00700 1 --2- 172.21.15.27:0/3618769946 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fe2700692a0 0x7fe27006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:36.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.188+0000 7fe28be00700 1 --2- 172.21.15.27:0/3618769946 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fe2700692a0 0x7fe27006b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00b3b0 tx=0x7fe27c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:36.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.193+0000 7fe283e00700 1 -- 172.21.15.27:0/3618769946 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fe2780572e0 con 0x7fe28c0ffd60 2024-04-16T05:40:36.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.464+0000 7fe291a00700 1 -- 172.21.15.27:0/3618769946 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe28c04bed0 con 0x7fe28c0ffd60 2024-04-16T05:40:36.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.464+0000 7fe283e00700 1 -- 172.21.15.27:0/3618769946 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1810 (secure 0 0 0) 0x7fe278011630 con 0x7fe28c0ffd60 2024-04-16T05:40:36.464 INFO:teuthology.orchestra.run.smithi027.stdout:e16 2024-04-16T05:40:36.464 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:40:36.464 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:40:36.464 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:40:36.464 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:40:36.464 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:40:36.464 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:40:36.465 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 15 2024-04-16T05:40:36.465 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:40:36.465 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:40:36.465 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:40:32.701649+0000 2024-04-16T05:40:36.465 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:40:36.465 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:40:36.465 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:40:36.465 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:40:36.465 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:40:36.465 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:40:36.465 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:40:36.465 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 46 2024-04-16T05:40:36.465 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:40:36.465 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:40:36.466 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:40:36.466 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=14462} 2024-04-16T05:40:36.466 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:40:36.466 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:40:36.466 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:40:36.466 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:40:36.466 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:40:36.466 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:40:36.466 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:40:36.466 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:40:36.466 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:40:36.467 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:40:36.467 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:40:36.467 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:40:36.467 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:40:36.467 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{-1:14474} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2065690722,v1:172.21.15.27:6827/2065690722] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:40:36.467 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:24263} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:40:36.467 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:24273} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:40:36.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.467+0000 7fe291a00700 1 -- 172.21.15.27:0/3618769946 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fe2700692a0 msgr2=0x7fe27006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:36.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.467+0000 7fe291a00700 1 --2- 172.21.15.27:0/3618769946 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fe2700692a0 0x7fe27006b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00b3b0 tx=0x7fe27c009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:36.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.467+0000 7fe291a00700 1 -- 172.21.15.27:0/3618769946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe28c0ffd60 msgr2=0x7fe28c10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:36.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.467+0000 7fe291a00700 1 --2- 172.21.15.27:0/3618769946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe28c0ffd60 0x7fe28c10de00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe27800c810 tx=0x7fe278004470 comp rx=0 tx=0).stop 2024-04-16T05:40:36.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.468+0000 7fe291a00700 1 -- 172.21.15.27:0/3618769946 shutdown_connections 2024-04-16T05:40:36.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.468+0000 7fe291a00700 1 --2- 172.21.15.27:0/3618769946 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fe2700692a0 0x7fe27006b750 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:36.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.468+0000 7fe291a00700 1 --2- 172.21.15.27:0/3618769946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe28c066ce0 0x7fe28c10d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:36.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.468+0000 7fe291a00700 1 --2- 172.21.15.27:0/3618769946 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe28c0ffd60 0x7fe28c10de00 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:36.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.468+0000 7fe291a00700 1 -- 172.21.15.27:0/3618769946 >> 172.21.15.27:0/3618769946 conn(0x7fe28c0f7290 msgr2=0x7fe28c0f76b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:36.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.468+0000 7fe291a00700 1 -- 172.21.15.27:0/3618769946 shutdown_connections 2024-04-16T05:40:36.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:36.468+0000 7fe291a00700 1 -- 172.21.15.27:0/3618769946 wait complete. 2024-04-16T05:40:36.469 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2024-04-16T05:40:36.856 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:36 smithi027 ceph-mon[29206]: pgmap v237: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-16T05:40:36.857 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:36 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/3618769946' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:40:37.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:36 smithi122 ceph-mon[35998]: pgmap v237: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-16T05:40:37.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:36 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/3618769946' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:40:37.780 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-04-16T05:40:38.676 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:40:39.005 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:38 smithi027 ceph-mon[29206]: pgmap v238: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-16T05:40:39.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:38 smithi122 ceph-mon[35998]: pgmap v238: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-16T05:40:40.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.851+0000 7f39f7e00700 1 -- 172.21.15.27:0/972402210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39f00709a0 msgr2=0x7f39f0070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:40.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.851+0000 7f39f7e00700 1 --2- 172.21.15.27:0/972402210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39f00709a0 0x7f39f0070e10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f39dc004940 tx=0x7f39dc009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:40.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.851+0000 7f39f7e00700 1 -- 172.21.15.27:0/972402210 shutdown_connections 2024-04-16T05:40:40.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.851+0000 7f39f7e00700 1 --2- 172.21.15.27:0/972402210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39f00709a0 0x7f39f0070e10 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:40.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.851+0000 7f39f7e00700 1 --2- 172.21.15.27:0/972402210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39f0071f50 0x7f39f0070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:40.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.851+0000 7f39f7e00700 1 -- 172.21.15.27:0/972402210 >> 172.21.15.27:0/972402210 conn(0x7f39f00f8530 msgr2=0x7f39f00fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:40.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.851+0000 7f39f7e00700 1 -- 172.21.15.27:0/972402210 shutdown_connections 2024-04-16T05:40:40.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.851+0000 7f39f7e00700 1 -- 172.21.15.27:0/972402210 wait complete. 2024-04-16T05:40:40.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.852+0000 7f39f7e00700 1 Processor -- start 2024-04-16T05:40:40.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.852+0000 7f39f7e00700 1 -- start start 2024-04-16T05:40:40.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.853+0000 7f39f7e00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39f00709a0 0x7f39f00fccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:40.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.853+0000 7f39f7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39f0071f50 0x7f39f00fd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:40.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.853+0000 7f39f7e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f00fd8d0 con 0x7f39f0071f50 2024-04-16T05:40:40.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.853+0000 7f39f7e00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39f00fe090 con 0x7f39f00709a0 2024-04-16T05:40:40.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.853+0000 7f39f4a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39f0071f50 0x7f39f00fd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:40.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.853+0000 7f39f4a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39f0071f50 0x7f39f00fd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45462/0 (socket says 172.21.15.27:45462) 2024-04-16T05:40:40.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.853+0000 7f39f4a00700 1 -- 172.21.15.27:0/1313697039 learned_addr learned my addr 172.21.15.27:0/1313697039 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:40:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.853+0000 7f39f5400700 1 --2- 172.21.15.27:0/1313697039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39f00709a0 0x7f39f00fccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.854+0000 7f39f4a00700 1 -- 172.21.15.27:0/1313697039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39f00709a0 msgr2=0x7f39f00fccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.854+0000 7f39f4a00700 1 --2- 172.21.15.27:0/1313697039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39f00709a0 0x7f39f00fccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.854+0000 7f39f4a00700 1 -- 172.21.15.27:0/1313697039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39dc005040 con 0x7f39f0071f50 2024-04-16T05:40:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.854+0000 7f39f4a00700 1 --2- 172.21.15.27:0/1313697039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39f0071f50 0x7f39f00fd1f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f39dc00b3b0 tx=0x7f39dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.854+0000 7f39ed600700 1 -- 172.21.15.27:0/1313697039 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39dc00e040 con 0x7f39f0071f50 2024-04-16T05:40:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.854+0000 7f39ed600700 1 -- 172.21.15.27:0/1313697039 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39dc004290 con 0x7f39f0071f50 2024-04-16T05:40:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.854+0000 7f39ed600700 1 -- 172.21.15.27:0/1313697039 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39dc01c630 con 0x7f39f0071f50 2024-04-16T05:40:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.854+0000 7f39f7e00700 1 -- 172.21.15.27:0/1313697039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39f00fe310 con 0x7f39f0071f50 2024-04-16T05:40:40.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.855+0000 7f39f7e00700 1 -- 172.21.15.27:0/1313697039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39f00fe8d0 con 0x7f39f0071f50 2024-04-16T05:40:40.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.856+0000 7f39f7e00700 1 -- 172.21.15.27:0/1313697039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39f01095c0 con 0x7f39f0071f50 2024-04-16T05:40:40.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.857+0000 7f39ed600700 1 -- 172.21.15.27:0/1313697039 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f39dc00bbf0 con 0x7f39f0071f50 2024-04-16T05:40:40.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.858+0000 7f39ed600700 1 --2- 172.21.15.27:0/1313697039 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f39d80692b0 0x7f39d806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:40.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.858+0000 7f39ed600700 1 -- 172.21.15.27:0/1313697039 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f39dc088660 con 0x7f39f0071f50 2024-04-16T05:40:40.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.858+0000 7f39f5400700 1 --2- 172.21.15.27:0/1313697039 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f39d80692b0 0x7f39d806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:40.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.859+0000 7f39f5400700 1 --2- 172.21.15.27:0/1313697039 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f39d80692b0 0x7f39d806b760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f39f00fca40 tx=0x7f39e4009620 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:40.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:40.863+0000 7f39ed600700 1 -- 172.21.15.27:0/1313697039 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f39dc054920 con 0x7f39f0071f50 2024-04-16T05:40:41.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:40 smithi122 ceph-mon[35998]: pgmap v239: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-16T05:40:41.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:40 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:41.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:40 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:41.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:40 smithi027 ceph-mon[29206]: pgmap v239: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-16T05:40:41.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:40 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:41.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:40 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:41.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:41.132+0000 7f39f7e00700 1 -- 172.21.15.27:0/1313697039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39f004bed0 con 0x7f39f0071f50 2024-04-16T05:40:41.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:41.133+0000 7f39ed600700 1 -- 172.21.15.27:0/1313697039 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4691 (secure 0 0 0) 0x7f39f004bed0 con 0x7f39f0071f50 2024-04-16T05:40:41.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:41.136+0000 7f39f7e00700 1 -- 172.21.15.27:0/1313697039 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f39d80692b0 msgr2=0x7f39d806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:41.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:41.136+0000 7f39f7e00700 1 --2- 172.21.15.27:0/1313697039 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f39d80692b0 0x7f39d806b760 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f39f00fca40 tx=0x7f39e4009620 comp rx=0 tx=0).stop 2024-04-16T05:40:41.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:41.136+0000 7f39f7e00700 1 -- 172.21.15.27:0/1313697039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39f0071f50 msgr2=0x7f39f00fd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:41.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:41.136+0000 7f39f7e00700 1 --2- 172.21.15.27:0/1313697039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39f0071f50 0x7f39f00fd1f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f39dc00b3b0 tx=0x7f39dc009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:41.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:41.136+0000 7f39f7e00700 1 -- 172.21.15.27:0/1313697039 shutdown_connections 2024-04-16T05:40:41.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:41.136+0000 7f39f7e00700 1 --2- 172.21.15.27:0/1313697039 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f39d80692b0 0x7f39d806b760 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:41.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:41.136+0000 7f39f7e00700 1 --2- 172.21.15.27:0/1313697039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f39f00709a0 0x7f39f00fccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:41.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:41.136+0000 7f39f7e00700 1 --2- 172.21.15.27:0/1313697039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39f0071f50 0x7f39f00fd1f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:41.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:41.136+0000 7f39f7e00700 1 -- 172.21.15.27:0/1313697039 >> 172.21.15.27:0/1313697039 conn(0x7f39f00f8530 msgr2=0x7f39f0102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:41.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:41.136+0000 7f39f7e00700 1 -- 172.21.15.27:0/1313697039 shutdown_connections 2024-04-16T05:40:41.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:41.136+0000 7f39f7e00700 1 -- 172.21.15.27:0/1313697039 wait complete. 2024-04-16T05:40:41.137 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2024-04-16T05:40:41.153 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:40:41.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:41 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1313697039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T05:40:41.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:41 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:40:41.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:41 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:40:41.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:41 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:41.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:41 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:40:42.010 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-04-16T05:40:42.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:41 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/1313697039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T05:40:42.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:41 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:40:42.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:41 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:40:42.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:41 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:42.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:41 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:40:42.541 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:40:43.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:42 smithi122 ceph-mon[35998]: pgmap v240: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-16T05:40:43.042 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:42 smithi027 ceph-mon[29206]: pgmap v240: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-16T05:40:45.007 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:44 smithi027 ceph-mon[29206]: pgmap v241: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.1 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-16T05:40:45.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:44 smithi122 ceph-mon[35998]: pgmap v241: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.1 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-16T05:40:45.884 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:45 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:45.884 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:45 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:45.884 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:45 smithi027 ceph-mon[29206]: pgmap v242: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-16T05:40:46.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.238+0000 7f8378600700 1 -- 172.21.15.27:0/988076973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8370071ea0 msgr2=0x7f83700703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:46.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.238+0000 7f8378600700 1 --2- 172.21.15.27:0/988076973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8370071ea0 0x7f83700703b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8364004940 tx=0x7f8364009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:46.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.239+0000 7f8378600700 1 -- 172.21.15.27:0/988076973 shutdown_connections 2024-04-16T05:40:46.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.239+0000 7f8378600700 1 --2- 172.21.15.27:0/988076973 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83700708f0 0x7f8370070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:46.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.239+0000 7f8378600700 1 --2- 172.21.15.27:0/988076973 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8370071ea0 0x7f83700703b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:46.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.239+0000 7f8378600700 1 -- 172.21.15.27:0/988076973 >> 172.21.15.27:0/988076973 conn(0x7f83700f8580 msgr2=0x7f83700fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:46.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.239+0000 7f8378600700 1 -- 172.21.15.27:0/988076973 shutdown_connections 2024-04-16T05:40:46.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.239+0000 7f8378600700 1 -- 172.21.15.27:0/988076973 wait complete. 2024-04-16T05:40:46.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.240+0000 7f8378600700 1 Processor -- start 2024-04-16T05:40:46.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.240+0000 7f8378600700 1 -- start start 2024-04-16T05:40:46.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.240+0000 7f8378600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83700708f0 0x7f8370113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:46.239 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.240+0000 7f8378600700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8370071ea0 0x7f83701144f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.240+0000 7f8378600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8370114af0 con 0x7f83700708f0 2024-04-16T05:40:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.240+0000 7f8378600700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8370114c60 con 0x7f8370071ea0 2024-04-16T05:40:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.240+0000 7f8375c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83700708f0 0x7f8370113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.241+0000 7f8375c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83700708f0 0x7f8370113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58580/0 (socket says 172.21.15.27:58580) 2024-04-16T05:40:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.241+0000 7f8375c00700 1 -- 172.21.15.27:0/2155056135 learned_addr learned my addr 172.21.15.27:0/2155056135 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:40:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.241+0000 7f8375200700 1 --2- 172.21.15.27:0/2155056135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8370071ea0 0x7f83701144f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.241+0000 7f8375c00700 1 -- 172.21.15.27:0/2155056135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8370071ea0 msgr2=0x7f83701144f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.241+0000 7f8375c00700 1 --2- 172.21.15.27:0/2155056135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8370071ea0 0x7f83701144f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.241+0000 7f8375c00700 1 -- 172.21.15.27:0/2155056135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8364005040 con 0x7f83700708f0 2024-04-16T05:40:46.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.241+0000 7f8375c00700 1 --2- 172.21.15.27:0/2155056135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83700708f0 0x7f8370113fb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f836400b970 tx=0x7f8364009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.242+0000 7f836d600700 1 -- 172.21.15.27:0/2155056135 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f836400e040 con 0x7f83700708f0 2024-04-16T05:40:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.242+0000 7f836d600700 1 -- 172.21.15.27:0/2155056135 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8364004290 con 0x7f83700708f0 2024-04-16T05:40:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.242+0000 7f8378600700 1 -- 172.21.15.27:0/2155056135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8370119660 con 0x7f83700708f0 2024-04-16T05:40:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.242+0000 7f8378600700 1 -- 172.21.15.27:0/2155056135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8370119c20 con 0x7f83700708f0 2024-04-16T05:40:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.242+0000 7f836d600700 1 -- 172.21.15.27:0/2155056135 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f836401c630 con 0x7f83700708f0 2024-04-16T05:40:46.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.243+0000 7f8378600700 1 -- 172.21.15.27:0/2155056135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83500052e0 con 0x7f83700708f0 2024-04-16T05:40:46.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.245+0000 7f836d600700 1 -- 172.21.15.27:0/2155056135 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f836400bbf0 con 0x7f83700708f0 2024-04-16T05:40:46.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.246+0000 7f836d600700 1 --2- 172.21.15.27:0/2155056135 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f83580692a0 0x7f835806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:46.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.246+0000 7f836d600700 1 -- 172.21.15.27:0/2155056135 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f83640890a0 con 0x7f83700708f0 2024-04-16T05:40:46.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.246+0000 7f8375200700 1 --2- 172.21.15.27:0/2155056135 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f83580692a0 0x7f835806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:46.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.247+0000 7f8375200700 1 --2- 172.21.15.27:0/2155056135 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f83580692a0 0x7f835806b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f835c009fd0 tx=0x7f835c009f20 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.250+0000 7f836d600700 1 -- 172.21.15.27:0/2155056135 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f8364058df0 con 0x7f83700708f0 2024-04-16T05:40:46.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:45 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:46.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:45 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:46.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:45 smithi122 ceph-mon[35998]: pgmap v242: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 0 B/s wr, 3 op/s 2024-04-16T05:40:46.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.566+0000 7f8378600700 1 -- 172.21.15.27:0/2155056135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f8350005860 con 0x7f83700708f0 2024-04-16T05:40:46.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.566+0000 7f836d600700 1 -- 172.21.15.27:0/2155056135 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v16) v1 ==== 78+0+99 (secure 0 0 0) 0x7f8364013e60 con 0x7f83700708f0 2024-04-16T05:40:46.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.569+0000 7f8378600700 1 -- 172.21.15.27:0/2155056135 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f83580692a0 msgr2=0x7f835806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.569+0000 7f8378600700 1 --2- 172.21.15.27:0/2155056135 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f83580692a0 0x7f835806b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f835c009fd0 tx=0x7f835c009f20 comp rx=0 tx=0).stop 2024-04-16T05:40:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.569+0000 7f8378600700 1 -- 172.21.15.27:0/2155056135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83700708f0 msgr2=0x7f8370113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.569+0000 7f8378600700 1 --2- 172.21.15.27:0/2155056135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83700708f0 0x7f8370113fb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f836400b970 tx=0x7f8364009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.569+0000 7f8378600700 1 -- 172.21.15.27:0/2155056135 shutdown_connections 2024-04-16T05:40:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.569+0000 7f8378600700 1 --2- 172.21.15.27:0/2155056135 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f83580692a0 0x7f835806b750 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.569+0000 7f8378600700 1 --2- 172.21.15.27:0/2155056135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83700708f0 0x7f8370113fb0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.569+0000 7f8378600700 1 --2- 172.21.15.27:0/2155056135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8370071ea0 0x7f83701144f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.569+0000 7f8378600700 1 -- 172.21.15.27:0/2155056135 >> 172.21.15.27:0/2155056135 conn(0x7f83700f8580 msgr2=0x7f83700f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.569+0000 7f8378600700 1 -- 172.21.15.27:0/2155056135 shutdown_connections 2024-04-16T05:40:46.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:46.569+0000 7f8378600700 1 -- 172.21.15.27:0/2155056135 wait complete. 2024-04-16T05:40:46.584 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:40:47.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:46 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/2155056135' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-16T05:40:47.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:46 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/2155056135' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-04-16T05:40:47.381 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-04-16T05:40:47.393 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T05:40:47.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.675+0000 7f148f800640 1 -- 172.21.15.27:0/3883676546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1488103a90 msgr2=0x7f1488103f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:47.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.675+0000 7f148f800640 1 --2- 172.21.15.27:0/3883676546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1488103a90 0x7f1488103f00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f14700048d0 tx=0x7f1470009da0 comp rx=0 tx=0).stop 2024-04-16T05:40:47.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.675+0000 7f148f800640 1 -- 172.21.15.27:0/3883676546 shutdown_connections 2024-04-16T05:40:47.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.675+0000 7f148f800640 1 --2- 172.21.15.27:0/3883676546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1488103a90 0x7f1488103f00 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:47.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.675+0000 7f148f800640 1 --2- 172.21.15.27:0/3883676546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14881027c0 0x7f1488102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:47.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.675+0000 7f148f800640 1 -- 172.21.15.27:0/3883676546 >> 172.21.15.27:0/3883676546 conn(0x7f1488071a50 msgr2=0x7f1488073e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:47.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.675+0000 7f148f800640 1 -- 172.21.15.27:0/3883676546 shutdown_connections 2024-04-16T05:40:47.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.675+0000 7f148f800640 1 -- 172.21.15.27:0/3883676546 wait complete. 2024-04-16T05:40:47.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.676+0000 7f148f800640 1 Processor -- start 2024-04-16T05:40:47.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.676+0000 7f148f800640 1 -- start start 2024-04-16T05:40:47.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.677+0000 7f148f800640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14881027c0 0x7f1488113270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:47.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.677+0000 7f148f800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1488103a90 0x7f14881137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:47.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.677+0000 7f148f800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1488113e00 con 0x7f1488103a90 2024-04-16T05:40:47.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.677+0000 7f148f800640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1488118800 con 0x7f14881027c0 2024-04-16T05:40:47.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.677+0000 7f1487e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1488103a90 0x7f14881137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:47.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.677+0000 7f1487e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1488103a90 0x7f14881137b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58600/0 (socket says 172.21.15.27:58600) 2024-04-16T05:40:47.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.677+0000 7f1487e00640 1 -- 172.21.15.27:0/3285575337 learned_addr learned my addr 172.21.15.27:0/3285575337 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:40:47.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.678+0000 7f148ce00640 1 --2- 172.21.15.27:0/3285575337 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14881027c0 0x7f1488113270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:47.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.678+0000 7f1487e00640 1 -- 172.21.15.27:0/3285575337 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14881027c0 msgr2=0x7f1488113270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:47.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.678+0000 7f1487e00640 1 --2- 172.21.15.27:0/3285575337 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14881027c0 0x7f1488113270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:47.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.678+0000 7f1487e00640 1 -- 172.21.15.27:0/3285575337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1470005040 con 0x7f1488103a90 2024-04-16T05:40:47.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.678+0000 7f1487e00640 1 --2- 172.21.15.27:0/3285575337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1488103a90 0x7f14881137b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1470000c00 tx=0x7f1470009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:47.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.679+0000 7f1484c00640 1 -- 172.21.15.27:0/3285575337 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14700094c0 con 0x7f1488103a90 2024-04-16T05:40:47.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.679+0000 7f1484c00640 1 -- 172.21.15.27:0/3285575337 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1470009620 con 0x7f1488103a90 2024-04-16T05:40:47.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.679+0000 7f1484c00640 1 -- 172.21.15.27:0/3285575337 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14700435f0 con 0x7f1488103a90 2024-04-16T05:40:47.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.679+0000 7f148f800640 1 -- 172.21.15.27:0/3285575337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1488118940 con 0x7f1488103a90 2024-04-16T05:40:47.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.679+0000 7f148f800640 1 -- 172.21.15.27:0/3285575337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1488118dd0 con 0x7f1488103a90 2024-04-16T05:40:47.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.680+0000 7f148f800640 1 -- 172.21.15.27:0/3285575337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1488064380 con 0x7f1488103a90 2024-04-16T05:40:47.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.681+0000 7f1484c00640 1 -- 172.21.15.27:0/3285575337 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f147002f070 con 0x7f1488103a90 2024-04-16T05:40:47.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.682+0000 7f1484c00640 1 --2- 172.21.15.27:0/3285575337 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f146c069210 0x7f146c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:47.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.682+0000 7f1484c00640 1 -- 172.21.15.27:0/3285575337 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f14700a9230 con 0x7f1488103a90 2024-04-16T05:40:47.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.682+0000 7f148ce00640 1 --2- 172.21.15.27:0/3285575337 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f146c069210 0x7f146c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:47.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.683+0000 7f148ce00640 1 --2- 172.21.15.27:0/3285575337 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f146c069210 0x7f146c06b6c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1478004e40 tx=0x7f1478009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:47.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.687+0000 7f1484c00640 1 -- 172.21.15.27:0/3285575337 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f1470075430 con 0x7f1488103a90 2024-04-16T05:40:47.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.891+0000 7f148f800640 1 -- 172.21.15.27:0/3285575337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f148810b320 con 0x7f1488103a90 2024-04-16T05:40:47.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.892+0000 7f1484c00640 1 -- 172.21.15.27:0/3285575337 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+14399 (secure 0 0 0) 0x7f1470078a50 con 0x7f1488103a90 2024-04-16T05:40:47.891 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:40:47.891 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":46,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","created":"2024-04-16T05:31:01.142297+0000","modified":"2024-04-16T05:40:29.686282+0000","last_up_change":"2024-04-16T05:38:30.698288+0000","last_in_change":"2024-04-16T05:38:07.718747+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T05:36:52.435427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-16T05:40:10.776546+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-16T05:40:11.582658+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"4dd7e294-f625-4492-8bd3-18beab34b6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1557847192},{"type":"v1","addr":"172.21.15.27:6803","nonce":1557847192}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1557847192},{"type":"v1","addr":"172.21.15.27:6805","nonce":1557847192}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1557847192},{"type":"v1","addr":"172.21.15.27:6809","nonce":1557847192}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1557847192},{"type":"v1","addr":"172.21.15.27:6807","nonce":1557847192}]},"public_addr":"172.21.15.27:6803/1557847192","cluster_addr":"172.21.15.27:6805/1557847192","heartbeat_back_addr":"172.21.15.27:6809/1557847192","heartbeat_front_addr":"172.21.15.27:6807/1557847192","state":["exists","up"]},{"osd":1,"uuid":"abec61aa-7a28-473b-b9d2-04e8d7de8a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2143934716},{"type":"v1","addr":"172.21.15.27:6811","nonce":2143934716}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2143934716},{"type":"v1","addr":"172.21.15.27:6813","nonce":2143934716}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2143934716},{"type":"v1","addr":"172.21.15.27:6817","nonce":2143934716}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2143934716},{"type":"v1","addr":"172.21.15.27:6815","nonce":2143934716}]},"public_addr":"172.21.15.27:6811/2143934716","cluster_addr":"172.21.15.27:6813/2143934716","heartbeat_back_addr":"172.21.15.27:6817/2143934716","heartbeat_front_addr":"172.21.15.27:6815/2143934716","state":["exists","up"]},{"osd":2,"uuid":"e50778e4-bf1f-45eb-aeab-05e8dad8fa89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":3370929967},{"type":"v1","addr":"172.21.15.27:6819","nonce":3370929967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":3370929967},{"type":"v1","addr":"172.21.15.27:6821","nonce":3370929967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":3370929967},{"type":"v1","addr":"172.21.15.27:6825","nonce":3370929967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":3370929967},{"type":"v1","addr":"172.21.15.27:6823","nonce":3370929967}]},"public_addr":"172.21.15.27:6819/3370929967","cluster_addr":"172.21.15.27:6821/3370929967","heartbeat_back_addr":"172.21.15.27:6825/3370929967","heartbeat_front_addr":"172.21.15.27:6823/3370929967","state":["exists","up"]},{"osd":3,"uuid":"8f55dd70-a9d7-4bc1-a79c-028e66571137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":186542313},{"type":"v1","addr":"172.21.15.122:6801","nonce":186542313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":186542313},{"type":"v1","addr":"172.21.15.122:6803","nonce":186542313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":186542313},{"type":"v1","addr":"172.21.15.122:6807","nonce":186542313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":186542313},{"type":"v1","addr":"172.21.15.122:6805","nonce":186542313}]},"public_addr":"172.21.15.122:6801/186542313","cluster_addr":"172.21.15.122:6803/186542313","heartbeat_back_addr":"172.21.15.122:6807/186542313","heartbeat_front_addr":"172.21.15.122:6805/186542313","state":["exists","up"]},{"osd":4,"uuid":"ba8734db-cb7a-4242-b354-8ff1f524f25a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2655596639},{"type":"v1","addr":"172.21.15.122:6809","nonce":2655596639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2655596639},{"type":"v1","addr":"172.21.15.122:6811","nonce":2655596639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2655596639},{"type":"v1","addr":"172.21.15.122:6815","nonce":2655596639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2655596639},{"type":"v1","addr":"172.21.15.122:6813","nonce":2655596639}]},"public_addr":"172.21.15.122:6809/2655596639","cluster_addr":"172.21.15.122:6811/2655596639","heartbeat_back_addr":"172.21.15.122:6815/2655596639","heartbeat_front_addr":"172.21.15.122:6813/2655596639","state":["exists","up"]},{"osd":5,"uuid":"6e2753a7-d194-4103-b586-bfef8cbefa2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":3490413277},{"type":"v1","addr":"172.21.15.122:6817","nonce":3490413277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":3490413277},{"type":"v1","addr":"172.21.15.122:6819","nonce":3490413277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":3490413277},{"type":"v1","addr":"172.21.15.122:6823","nonce":3490413277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":3490413277},{"type":"v1","addr":"172.21.15.122:6821","nonce":3490413277}]},"public_addr":"172.21.15.122:6817/3490413277","cluster_addr":"172.21.15.122:6819/3490413277","heartbeat_back_addr":"172.21.15.122:6823/3490413277","heartbeat_front_addr":"172.21.15.122:6821/3490413277","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:35:28.179537+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:36:08.890364+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:36:48.694394+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:37:21.969883+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:37:55.840648+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:38:29.596986+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:6827/1980301393":"2024-04-17T05:40:29.686229+0000","172.21.15.27:0/3775595668":"2024-04-17T05:32:17.529651+0000","172.21.15.27:6826/1980301393":"2024-04-17T05:40:29.686229+0000","172.21.15.27:0/3713909498":"2024-04-17T05:31:34.994394+0000","172.21.15.27:6800/2779081505":"2024-04-17T05:32:17.529651+0000","172.21.15.27:6801/2173136120":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/3170976796":"2024-04-17T05:31:34.994394+0000","172.21.15.27:6800/2700787222":"2024-04-17T05:33:49.465983+0000","172.21.15.27:0/1435069943":"2024-04-17T05:31:34.994394+0000","172.21.15.27:6800/2173136120":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/2242461930":"2024-04-17T05:33:49.465983+0000","172.21.15.27:0/3086153578":"2024-04-17T05:32:17.529651+0000","172.21.15.27:6801/2700787222":"2024-04-17T05:33:49.465983+0000","172.21.15.27:0/602364106":"2024-04-17T05:33:49.465983+0000","172.21.15.27:6801/2779081505":"2024-04-17T05:32:17.529651+0000","172.21.15.27:0/4258676557":"2024-04-17T05:32:17.529651+0000","172.21.15.27:0/1860000032":"2024-04-17T05:33:49.465983+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T05:40:47.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.895+0000 7f148f800640 1 -- 172.21.15.27:0/3285575337 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f146c069210 msgr2=0x7f146c06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:47.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.895+0000 7f148f800640 1 --2- 172.21.15.27:0/3285575337 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f146c069210 0x7f146c06b6c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1478004e40 tx=0x7f1478009040 comp rx=0 tx=0).stop 2024-04-16T05:40:47.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.895+0000 7f148f800640 1 -- 172.21.15.27:0/3285575337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1488103a90 msgr2=0x7f14881137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:47.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.895+0000 7f148f800640 1 --2- 172.21.15.27:0/3285575337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1488103a90 0x7f14881137b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1470000c00 tx=0x7f1470009da0 comp rx=0 tx=0).stop 2024-04-16T05:40:47.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.895+0000 7f148f800640 1 -- 172.21.15.27:0/3285575337 shutdown_connections 2024-04-16T05:40:47.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.895+0000 7f148f800640 1 --2- 172.21.15.27:0/3285575337 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f146c069210 0x7f146c06b6c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:47.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.895+0000 7f148f800640 1 --2- 172.21.15.27:0/3285575337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1488103a90 0x7f14881137b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:47.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.895+0000 7f148f800640 1 --2- 172.21.15.27:0/3285575337 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f14881027c0 0x7f1488113270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:47.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.895+0000 7f148f800640 1 -- 172.21.15.27:0/3285575337 >> 172.21.15.27:0/3285575337 conn(0x7f1488071a50 msgr2=0x7f14880730d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:47.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.895+0000 7f148f800640 1 -- 172.21.15.27:0/3285575337 shutdown_connections 2024-04-16T05:40:47.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:47.895+0000 7f148f800640 1 -- 172.21.15.27:0/3285575337 wait complete. 2024-04-16T05:40:47.912 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T05:36:52.435427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-16T05:40:10.776546+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-16T05:40:11.582658+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-16T05:40:47.912 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T05:40:48.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.125+0000 7fb911200640 1 -- 172.21.15.27:0/3741378919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb90c0fe3d0 msgr2=0x7fb90c0fe7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:48.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.125+0000 7fb911200640 1 --2- 172.21.15.27:0/3741378919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb90c0fe3d0 0x7fb90c0fe7e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc0048d0 tx=0x7fb8fc009da0 comp rx=0 tx=0).stop 2024-04-16T05:40:48.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.125+0000 7fb911200640 1 -- 172.21.15.27:0/3741378919 shutdown_connections 2024-04-16T05:40:48.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.125+0000 7fb911200640 1 --2- 172.21.15.27:0/3741378919 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb90c0ff6a0 0x7fb90c0ffb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:48.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.125+0000 7fb911200640 1 --2- 172.21.15.27:0/3741378919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb90c0fe3d0 0x7fb90c0fe7e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:48.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.125+0000 7fb911200640 1 -- 172.21.15.27:0/3741378919 >> 172.21.15.27:0/3741378919 conn(0x7fb90c0f9c80 msgr2=0x7fb90c0fc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:48.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.126+0000 7fb911200640 1 -- 172.21.15.27:0/3741378919 shutdown_connections 2024-04-16T05:40:48.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.126+0000 7fb911200640 1 -- 172.21.15.27:0/3741378919 wait complete. 2024-04-16T05:40:48.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.126+0000 7fb911200640 1 Processor -- start 2024-04-16T05:40:48.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.127+0000 7fb911200640 1 -- start start 2024-04-16T05:40:48.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.127+0000 7fb911200640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb90c0fe3d0 0x7fb90c10eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:48.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.127+0000 7fb911200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb90c0ff6a0 0x7fb90c10f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:48.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.127+0000 7fb911200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c10fa70 con 0x7fb90c0ff6a0 2024-04-16T05:40:48.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.127+0000 7fb911200640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c114470 con 0x7fb90c0fe3d0 2024-04-16T05:40:48.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.127+0000 7fb901400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb90c0ff6a0 0x7fb90c10f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:48.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.127+0000 7fb901400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb90c0ff6a0 0x7fb90c10f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58620/0 (socket says 172.21.15.27:58620) 2024-04-16T05:40:48.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.127+0000 7fb901400640 1 -- 172.21.15.27:0/548602332 learned_addr learned my addr 172.21.15.27:0/548602332 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:40:48.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.128+0000 7fb901400640 1 -- 172.21.15.27:0/548602332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb90c0fe3d0 msgr2=0x7fb90c10eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:48.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.128+0000 7fb909e00640 1 --2- 172.21.15.27:0/548602332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb90c0fe3d0 0x7fb90c10eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:48.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.128+0000 7fb901400640 1 --2- 172.21.15.27:0/548602332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb90c0fe3d0 0x7fb90c10eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:48.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.128+0000 7fb901400640 1 -- 172.21.15.27:0/548602332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8fc005040 con 0x7fb90c0ff6a0 2024-04-16T05:40:48.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.128+0000 7fb901400640 1 --2- 172.21.15.27:0/548602332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb90c0ff6a0 0x7fb90c10f420 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4002a50 tx=0x7fb8f4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:48.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.128+0000 7fb902a00640 1 -- 172.21.15.27:0/548602332 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8f4007c30 con 0x7fb90c0ff6a0 2024-04-16T05:40:48.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.128+0000 7fb902a00640 1 -- 172.21.15.27:0/548602332 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8f400f040 con 0x7fb90c0ff6a0 2024-04-16T05:40:48.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.128+0000 7fb911200640 1 -- 172.21.15.27:0/548602332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90c114610 con 0x7fb90c0ff6a0 2024-04-16T05:40:48.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.128+0000 7fb902a00640 1 -- 172.21.15.27:0/548602332 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8f400db50 con 0x7fb90c0ff6a0 2024-04-16T05:40:48.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.129+0000 7fb911200640 1 -- 172.21.15.27:0/548602332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb90c114b00 con 0x7fb90c0ff6a0 2024-04-16T05:40:48.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.130+0000 7fb911200640 1 -- 172.21.15.27:0/548602332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb90c04bcc0 con 0x7fb90c0ff6a0 2024-04-16T05:40:48.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.131+0000 7fb902a00640 1 -- 172.21.15.27:0/548602332 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fb8f401c400 con 0x7fb90c0ff6a0 2024-04-16T05:40:48.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.131+0000 7fb902a00640 1 --2- 172.21.15.27:0/548602332 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb8f0069210 0x7fb8f006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:48.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.132+0000 7fb902a00640 1 -- 172.21.15.27:0/548602332 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fb8f4086b80 con 0x7fb90c0ff6a0 2024-04-16T05:40:48.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.132+0000 7fb909e00640 1 --2- 172.21.15.27:0/548602332 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb8f0069210 0x7fb8f006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:48.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.133+0000 7fb909e00640 1 --2- 172.21.15.27:0/548602332 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb8f0069210 0x7fb8f006b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc02bda0 tx=0x7fb8fc009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:48.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.136+0000 7fb902a00640 1 -- 172.21.15.27:0/548602332 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb8f404d250 con 0x7fb90c0ff6a0 2024-04-16T05:40:48.173 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:47 smithi122 ceph-mon[35998]: pgmap v243: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-16T05:40:48.245 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:47 smithi027 ceph-mon[29206]: pgmap v243: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-16T05:40:48.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.340+0000 7fb911200640 1 -- 172.21.15.27:0/548602332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb90c114d50 con 0x7fb90c0ff6a0 2024-04-16T05:40:48.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.341+0000 7fb902a00640 1 -- 172.21.15.27:0/548602332 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb8f4052d40 con 0x7fb90c0ff6a0 2024-04-16T05:40:48.339 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-16T05:40:48.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.343+0000 7fb911200640 1 -- 172.21.15.27:0/548602332 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb8f0069210 msgr2=0x7fb8f006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:48.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.343+0000 7fb911200640 1 --2- 172.21.15.27:0/548602332 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb8f0069210 0x7fb8f006b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc02bda0 tx=0x7fb8fc009da0 comp rx=0 tx=0).stop 2024-04-16T05:40:48.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.344+0000 7fb911200640 1 -- 172.21.15.27:0/548602332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb90c0ff6a0 msgr2=0x7fb90c10f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:48.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.344+0000 7fb911200640 1 --2- 172.21.15.27:0/548602332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb90c0ff6a0 0x7fb90c10f420 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4002a50 tx=0x7fb8f4004600 comp rx=0 tx=0).stop 2024-04-16T05:40:48.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.344+0000 7fb911200640 1 -- 172.21.15.27:0/548602332 shutdown_connections 2024-04-16T05:40:48.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.344+0000 7fb911200640 1 --2- 172.21.15.27:0/548602332 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb8f0069210 0x7fb8f006b6c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:48.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.344+0000 7fb911200640 1 --2- 172.21.15.27:0/548602332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb90c0ff6a0 0x7fb90c10f420 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:48.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.344+0000 7fb911200640 1 --2- 172.21.15.27:0/548602332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb90c0fe3d0 0x7fb90c10eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:48.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.344+0000 7fb911200640 1 -- 172.21.15.27:0/548602332 >> 172.21.15.27:0/548602332 conn(0x7fb90c0f9c80 msgr2=0x7fb90c0fb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:48.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.344+0000 7fb911200640 1 -- 172.21.15.27:0/548602332 shutdown_connections 2024-04-16T05:40:48.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.344+0000 7fb911200640 1 -- 172.21.15.27:0/548602332 wait complete. 2024-04-16T05:40:48.361 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-16T05:40:48.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.558+0000 7f8f4fa00640 1 -- 172.21.15.27:0/1266583352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f48101770 msgr2=0x7f8f48103b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:48.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.558+0000 7f8f4fa00640 1 --2- 172.21.15.27:0/1266583352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f48101770 0x7f8f48103b50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8f340048d0 tx=0x7f8f34009da0 comp rx=0 tx=0).stop 2024-04-16T05:40:48.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.558+0000 7f8f4fa00640 1 -- 172.21.15.27:0/1266583352 shutdown_connections 2024-04-16T05:40:48.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.558+0000 7f8f4fa00640 1 --2- 172.21.15.27:0/1266583352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f48101770 0x7f8f48103b50 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:48.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.558+0000 7f8f4fa00640 1 --2- 172.21.15.27:0/1266583352 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f48066120 0x7f8f48101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:48.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.558+0000 7f8f4fa00640 1 -- 172.21.15.27:0/1266583352 >> 172.21.15.27:0/1266583352 conn(0x7f8f480f6880 msgr2=0x7f8f480f8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:48.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.558+0000 7f8f4fa00640 1 -- 172.21.15.27:0/1266583352 shutdown_connections 2024-04-16T05:40:48.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.558+0000 7f8f4fa00640 1 -- 172.21.15.27:0/1266583352 wait complete. 2024-04-16T05:40:48.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.559+0000 7f8f4fa00640 1 Processor -- start 2024-04-16T05:40:48.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.559+0000 7f8f4fa00640 1 -- start start 2024-04-16T05:40:48.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.559+0000 7f8f4fa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f48066120 0x7f8f4810ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:48.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.559+0000 7f8f4fa00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f48101770 0x7f8f4810f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:48.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.559+0000 7f8f4fa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4810fab0 con 0x7f8f48066120 2024-04-16T05:40:48.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.559+0000 7f8f4fa00640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f481144b0 con 0x7f8f48101770 2024-04-16T05:40:48.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.559+0000 7f8f4d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f48066120 0x7f8f4810ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:48.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.560+0000 7f8f4d000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f48066120 0x7f8f4810ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58632/0 (socket says 172.21.15.27:58632) 2024-04-16T05:40:48.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.560+0000 7f8f4d000640 1 -- 172.21.15.27:0/994642784 learned_addr learned my addr 172.21.15.27:0/994642784 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:40:48.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.560+0000 7f8f4d000640 1 -- 172.21.15.27:0/994642784 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f48101770 msgr2=0x7f8f4810f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:48.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.560+0000 7f8f4d000640 1 --2- 172.21.15.27:0/994642784 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f48101770 0x7f8f4810f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:48.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.560+0000 7f8f4d000640 1 -- 172.21.15.27:0/994642784 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f34005040 con 0x7f8f48066120 2024-04-16T05:40:48.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.560+0000 7f8f4d000640 1 --2- 172.21.15.27:0/994642784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f48066120 0x7f8f4810ef20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c00c7e0 tx=0x7f8f3c0043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:48.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.560+0000 7f8f44c00640 1 -- 172.21.15.27:0/994642784 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f3c00a820 con 0x7f8f48066120 2024-04-16T05:40:48.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.560+0000 7f8f44c00640 1 -- 172.21.15.27:0/994642784 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f3c00ae50 con 0x7f8f48066120 2024-04-16T05:40:48.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.560+0000 7f8f44c00640 1 -- 172.21.15.27:0/994642784 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f3c0113c0 con 0x7f8f48066120 2024-04-16T05:40:48.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.560+0000 7f8f4fa00640 1 -- 172.21.15.27:0/994642784 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f48114650 con 0x7f8f48066120 2024-04-16T05:40:48.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.561+0000 7f8f4fa00640 1 -- 172.21.15.27:0/994642784 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f48114b40 con 0x7f8f48066120 2024-04-16T05:40:48.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.562+0000 7f8f4fa00640 1 -- 172.21.15.27:0/994642784 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f4804bcc0 con 0x7f8f48066120 2024-04-16T05:40:48.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.563+0000 7f8f44c00640 1 -- 172.21.15.27:0/994642784 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f8f3c012810 con 0x7f8f48066120 2024-04-16T05:40:48.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.563+0000 7f8f44c00640 1 --2- 172.21.15.27:0/994642784 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f8f30069220 0x7f8f3006b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:48.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.564+0000 7f8f44c00640 1 -- 172.21.15.27:0/994642784 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f8f3c0873b0 con 0x7f8f48066120 2024-04-16T05:40:48.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.564+0000 7f8f47e00640 1 --2- 172.21.15.27:0/994642784 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f8f30069220 0x7f8f3006b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:48.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.564+0000 7f8f47e00640 1 --2- 172.21.15.27:0/994642784 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f8f30069220 0x7f8f3006b6d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8f340317e0 tx=0x7f8f34009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:48.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.568+0000 7f8f44c00640 1 -- 172.21.15.27:0/994642784 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f8f3c057100 con 0x7f8f48066120 2024-04-16T05:40:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.755+0000 7f8f4fa00640 1 -- 172.21.15.27:0/994642784 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f8f48114ee0 con 0x7f8f48066120 2024-04-16T05:40:48.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.755+0000 7f8f44c00640 1 -- 172.21.15.27:0/994642784 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f8f3c056c90 con 0x7f8f48066120 2024-04-16T05:40:48.754 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-16T05:40:48.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.757+0000 7f8f4fa00640 1 -- 172.21.15.27:0/994642784 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f8f30069220 msgr2=0x7f8f3006b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:48.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.758+0000 7f8f4fa00640 1 --2- 172.21.15.27:0/994642784 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f8f30069220 0x7f8f3006b6d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8f340317e0 tx=0x7f8f34009da0 comp rx=0 tx=0).stop 2024-04-16T05:40:48.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.758+0000 7f8f4fa00640 1 -- 172.21.15.27:0/994642784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f48066120 msgr2=0x7f8f4810ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:48.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.758+0000 7f8f4fa00640 1 --2- 172.21.15.27:0/994642784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f48066120 0x7f8f4810ef20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c00c7e0 tx=0x7f8f3c0043c0 comp rx=0 tx=0).stop 2024-04-16T05:40:48.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.758+0000 7f8f4fa00640 1 -- 172.21.15.27:0/994642784 shutdown_connections 2024-04-16T05:40:48.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.758+0000 7f8f4fa00640 1 --2- 172.21.15.27:0/994642784 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f8f30069220 0x7f8f3006b6d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:48.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.758+0000 7f8f4fa00640 1 --2- 172.21.15.27:0/994642784 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f48101770 0x7f8f4810f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:48.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.758+0000 7f8f4fa00640 1 --2- 172.21.15.27:0/994642784 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f48066120 0x7f8f4810ef20 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:48.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.758+0000 7f8f4fa00640 1 -- 172.21.15.27:0/994642784 >> 172.21.15.27:0/994642784 conn(0x7f8f480f6880 msgr2=0x7f8f480f6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:48.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.758+0000 7f8f4fa00640 1 -- 172.21.15.27:0/994642784 shutdown_connections 2024-04-16T05:40:48.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.758+0000 7f8f4fa00640 1 -- 172.21.15.27:0/994642784 wait complete. 2024-04-16T05:40:48.777 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-16T05:40:48.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.974+0000 7fa23fe00640 1 -- 172.21.15.27:0/2459136567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa240065040 msgr2=0x7fa2400fec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:48.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.974+0000 7fa23fe00640 1 --2- 172.21.15.27:0/2459136567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa240065040 0x7fa2400fec10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa2300048d0 tx=0x7fa230009da0 comp rx=0 tx=0).stop 2024-04-16T05:40:48.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.974+0000 7fa23fe00640 1 -- 172.21.15.27:0/2459136567 shutdown_connections 2024-04-16T05:40:48.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.974+0000 7fa23fe00640 1 --2- 172.21.15.27:0/2459136567 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2400ff150 0x7fa240101530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:48.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.974+0000 7fa23fe00640 1 --2- 172.21.15.27:0/2459136567 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa240065040 0x7fa2400fec10 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:48.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.974+0000 7fa23fe00640 1 -- 172.21.15.27:0/2459136567 >> 172.21.15.27:0/2459136567 conn(0x7fa2400f6720 msgr2=0x7fa2400f8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:48.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.974+0000 7fa23fe00640 1 -- 172.21.15.27:0/2459136567 shutdown_connections 2024-04-16T05:40:48.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.974+0000 7fa23fe00640 1 -- 172.21.15.27:0/2459136567 wait complete. 2024-04-16T05:40:48.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.975+0000 7fa23fe00640 1 Processor -- start 2024-04-16T05:40:48.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.975+0000 7fa23fe00640 1 -- start start 2024-04-16T05:40:48.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.975+0000 7fa23fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa240065040 0x7fa24010edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:48.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.975+0000 7fa23fe00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2400ff150 0x7fa24010f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:48.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.975+0000 7fa23fe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa24010f9e0 con 0x7fa240065040 2024-04-16T05:40:48.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.975+0000 7fa23fe00640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa240114350 con 0x7fa2400ff150 2024-04-16T05:40:48.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.976+0000 7fa23ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa240065040 0x7fa24010edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:48.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.976+0000 7fa23ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa240065040 0x7fa24010edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58664/0 (socket says 172.21.15.27:58664) 2024-04-16T05:40:48.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.976+0000 7fa23ea00640 1 -- 172.21.15.27:0/3781921552 learned_addr learned my addr 172.21.15.27:0/3781921552 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:40:48.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.976+0000 7fa23e000640 1 --2- 172.21.15.27:0/3781921552 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2400ff150 0x7fa24010f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:48.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.976+0000 7fa23ea00640 1 -- 172.21.15.27:0/3781921552 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2400ff150 msgr2=0x7fa24010f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:48.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.976+0000 7fa23ea00640 1 --2- 172.21.15.27:0/3781921552 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2400ff150 0x7fa24010f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:48.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.976+0000 7fa23ea00640 1 -- 172.21.15.27:0/3781921552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa230005040 con 0x7fa240065040 2024-04-16T05:40:48.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.976+0000 7fa23ea00640 1 --2- 172.21.15.27:0/3781921552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa240065040 0x7fa24010edc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa230000c00 tx=0x7fa230009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:48.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.977+0000 7fa236a00640 1 -- 172.21.15.27:0/3781921552 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2300094c0 con 0x7fa240065040 2024-04-16T05:40:48.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.977+0000 7fa236a00640 1 -- 172.21.15.27:0/3781921552 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa230009620 con 0x7fa240065040 2024-04-16T05:40:48.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.977+0000 7fa23fe00640 1 -- 172.21.15.27:0/3781921552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2401144f0 con 0x7fa240065040 2024-04-16T05:40:48.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.977+0000 7fa236a00640 1 -- 172.21.15.27:0/3781921552 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa2300435f0 con 0x7fa240065040 2024-04-16T05:40:48.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.977+0000 7fa23fe00640 1 -- 172.21.15.27:0/3781921552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa240114a10 con 0x7fa240065040 2024-04-16T05:40:48.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.978+0000 7fa23fe00640 1 -- 172.21.15.27:0/3781921552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa24004bcc0 con 0x7fa240065040 2024-04-16T05:40:48.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.979+0000 7fa236a00640 1 -- 172.21.15.27:0/3781921552 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fa23002f070 con 0x7fa240065040 2024-04-16T05:40:48.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.980+0000 7fa236a00640 1 --2- 172.21.15.27:0/3781921552 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa224069220 0x7fa22406b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:48.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.980+0000 7fa236a00640 1 -- 172.21.15.27:0/3781921552 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa2300a9220 con 0x7fa240065040 2024-04-16T05:40:48.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.980+0000 7fa23e000640 1 --2- 172.21.15.27:0/3781921552 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa224069220 0x7fa22406b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:48.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.981+0000 7fa23e000640 1 --2- 172.21.15.27:0/3781921552 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa224069220 0x7fa22406b6d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa2401102c0 tx=0x7fa22c008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:48.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:48.985+0000 7fa236a00640 1 -- 172.21.15.27:0/3781921552 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa230075420 con 0x7fa240065040 2024-04-16T05:40:49.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:48 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/3285575337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T05:40:49.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:48 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/548602332' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T05:40:49.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:48 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/994642784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T05:40:49.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.174+0000 7fa23fe00640 1 -- 172.21.15.27:0/3781921552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fa2400f81c0 con 0x7fa240065040 2024-04-16T05:40:49.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.175+0000 7fa236a00640 1 -- 172.21.15.27:0/3781921552 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7fa230078a40 con 0x7fa240065040 2024-04-16T05:40:49.174 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-16T05:40:49.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.178+0000 7fa23fe00640 1 -- 172.21.15.27:0/3781921552 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa224069220 msgr2=0x7fa22406b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:49.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.178+0000 7fa23fe00640 1 --2- 172.21.15.27:0/3781921552 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa224069220 0x7fa22406b6d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa2401102c0 tx=0x7fa22c008040 comp rx=0 tx=0).stop 2024-04-16T05:40:49.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.178+0000 7fa23fe00640 1 -- 172.21.15.27:0/3781921552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa240065040 msgr2=0x7fa24010edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:49.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.178+0000 7fa23fe00640 1 --2- 172.21.15.27:0/3781921552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa240065040 0x7fa24010edc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa230000c00 tx=0x7fa230009da0 comp rx=0 tx=0).stop 2024-04-16T05:40:49.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.178+0000 7fa23fe00640 1 -- 172.21.15.27:0/3781921552 shutdown_connections 2024-04-16T05:40:49.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.178+0000 7fa23fe00640 1 --2- 172.21.15.27:0/3781921552 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa224069220 0x7fa22406b6d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:49.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.178+0000 7fa23fe00640 1 --2- 172.21.15.27:0/3781921552 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2400ff150 0x7fa24010f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:49.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.178+0000 7fa23fe00640 1 --2- 172.21.15.27:0/3781921552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa240065040 0x7fa24010edc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:49.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.178+0000 7fa23fe00640 1 -- 172.21.15.27:0/3781921552 >> 172.21.15.27:0/3781921552 conn(0x7fa2400f6720 msgr2=0x7fa2400ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:49.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.179+0000 7fa23fe00640 1 -- 172.21.15.27:0/3781921552 shutdown_connections 2024-04-16T05:40:49.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.179+0000 7fa23fe00640 1 -- 172.21.15.27:0/3781921552 wait complete. 2024-04-16T05:40:49.197 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T05:40:49.210 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:48 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/3285575337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T05:40:49.210 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:48 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/548602332' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T05:40:49.210 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:48 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/994642784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T05:40:49.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.395+0000 7f8105400640 1 -- 172.21.15.27:0/2985037299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8100065040 msgr2=0x7f8100100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:49.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.395+0000 7f8105400640 1 --2- 172.21.15.27:0/2985037299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8100065040 0x7f8100100fb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f80f00048d0 tx=0x7f80f0009da0 comp rx=0 tx=0).stop 2024-04-16T05:40:49.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.395+0000 7f8105400640 1 -- 172.21.15.27:0/2985037299 shutdown_connections 2024-04-16T05:40:49.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.395+0000 7f8105400640 1 --2- 172.21.15.27:0/2985037299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f81001014f0 0x7f81001038d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:49.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.395+0000 7f8105400640 1 --2- 172.21.15.27:0/2985037299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8100065040 0x7f8100100fb0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:49.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.395+0000 7f8105400640 1 -- 172.21.15.27:0/2985037299 >> 172.21.15.27:0/2985037299 conn(0x7f81000f6680 msgr2=0x7f81000f8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:49.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.396+0000 7f8105400640 1 -- 172.21.15.27:0/2985037299 shutdown_connections 2024-04-16T05:40:49.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.396+0000 7f8105400640 1 -- 172.21.15.27:0/2985037299 wait complete. 2024-04-16T05:40:49.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.396+0000 7f8105400640 1 Processor -- start 2024-04-16T05:40:49.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.396+0000 7f8105400640 1 -- start start 2024-04-16T05:40:49.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.397+0000 7f8105400640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8100065040 0x7f810010edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:49.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.397+0000 7f8105400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81001014f0 0x7f810010f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:49.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.397+0000 7f8105400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810010f9d0 con 0x7f81001014f0 2024-04-16T05:40:49.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.397+0000 7f8105400640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8100114300 con 0x7f8100065040 2024-04-16T05:40:49.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.397+0000 7f80ff400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81001014f0 0x7f810010f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:49.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.397+0000 7f80ff400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81001014f0 0x7f810010f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58676/0 (socket says 172.21.15.27:58676) 2024-04-16T05:40:49.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.397+0000 7f80ff400640 1 -- 172.21.15.27:0/4263617434 learned_addr learned my addr 172.21.15.27:0/4263617434 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:40:49.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.397+0000 7f80ffe00640 1 --2- 172.21.15.27:0/4263617434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8100065040 0x7f810010edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:49.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.397+0000 7f80ff400640 1 -- 172.21.15.27:0/4263617434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8100065040 msgr2=0x7f810010edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:49.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.397+0000 7f80ff400640 1 --2- 172.21.15.27:0/4263617434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8100065040 0x7f810010edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:49.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.397+0000 7f80ff400640 1 -- 172.21.15.27:0/4263617434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f0005040 con 0x7f81001014f0 2024-04-16T05:40:49.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.398+0000 7f80ff400640 1 --2- 172.21.15.27:0/4263617434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81001014f0 0x7f810010f2f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00b790 tx=0x7f80ec0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:49.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.398+0000 7f80f7e00640 1 -- 172.21.15.27:0/4263617434 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80ec00c700 con 0x7f81001014f0 2024-04-16T05:40:49.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.398+0000 7f80f7e00640 1 -- 172.21.15.27:0/4263617434 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f80ec00cd30 con 0x7f81001014f0 2024-04-16T05:40:49.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.398+0000 7f8105400640 1 -- 172.21.15.27:0/4263617434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8100114500 con 0x7f81001014f0 2024-04-16T05:40:49.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.398+0000 7f80f7e00640 1 -- 172.21.15.27:0/4263617434 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80ec013370 con 0x7f81001014f0 2024-04-16T05:40:49.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.398+0000 7f8105400640 1 -- 172.21.15.27:0/4263617434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8100114a20 con 0x7f81001014f0 2024-04-16T05:40:49.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.399+0000 7f8105400640 1 -- 172.21.15.27:0/4263617434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f810004bcc0 con 0x7f81001014f0 2024-04-16T05:40:49.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.401+0000 7f80f7e00640 1 -- 172.21.15.27:0/4263617434 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f80ec012070 con 0x7f81001014f0 2024-04-16T05:40:49.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.402+0000 7f80f7e00640 1 --2- 172.21.15.27:0/4263617434 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f80e4069220 0x7f80e406b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:49.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.402+0000 7f80f7e00640 1 -- 172.21.15.27:0/4263617434 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f80ec086e80 con 0x7f81001014f0 2024-04-16T05:40:49.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.402+0000 7f80ffe00640 1 --2- 172.21.15.27:0/4263617434 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f80e4069220 0x7f80e406b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:49.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.403+0000 7f80ffe00640 1 --2- 172.21.15.27:0/4263617434 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f80e4069220 0x7f80e406b6d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f80f0000c00 tx=0x7f80f0009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:49.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.406+0000 7f80f7e00640 1 -- 172.21.15.27:0/4263617434 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f80ec0530e0 con 0x7f81001014f0 2024-04-16T05:40:49.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.652+0000 7f8105400640 1 -- 172.21.15.27:0/4263617434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8100114ca0 con 0x7f81001014f0 2024-04-16T05:40:49.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.653+0000 7f80f7e00640 1 -- 172.21.15.27:0/4263617434 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4691 (secure 0 0 0) 0x7f80ec056700 con 0x7f81001014f0 2024-04-16T05:40:49.652 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:40:49.652 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14474,"name":"cephfs.smithi027.hkhpuq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/2065690722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2065690722},{"type":"v1","addr":"172.21.15.27:6827","nonce":2065690722}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24263,"name":"cephfs.smithi122.lojjdp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.122:6825/194115669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":194115669},{"type":"v1","addr":"172.21.15.122:6825","nonce":194115669}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":24273,"name":"cephfs.smithi122.rulewi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1078307975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1078307975},{"type":"v1","addr":"172.21.15.122:6827","nonce":1078307975}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T05:40:11.858650+0000","modified":"2024-04-16T05:40:32.701649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14462},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14462":{"gid":14462,"name":"cephfs.smithi027.ejoytk","rank":0,"incarnation":12,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6829/4115272713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4115272713},{"type":"v1","addr":"172.21.15.27:6829","nonce":4115272713}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-04-16T05:40:49.652 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2024-04-16T05:40:49.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.656+0000 7f8105400640 1 -- 172.21.15.27:0/4263617434 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f80e4069220 msgr2=0x7f80e406b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:49.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.656+0000 7f8105400640 1 --2- 172.21.15.27:0/4263617434 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f80e4069220 0x7f80e406b6d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f80f0000c00 tx=0x7f80f0009da0 comp rx=0 tx=0).stop 2024-04-16T05:40:49.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.656+0000 7f8105400640 1 -- 172.21.15.27:0/4263617434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81001014f0 msgr2=0x7f810010f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:49.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.656+0000 7f8105400640 1 --2- 172.21.15.27:0/4263617434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81001014f0 0x7f810010f2f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00b790 tx=0x7f80ec0091d0 comp rx=0 tx=0).stop 2024-04-16T05:40:49.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.656+0000 7f8105400640 1 -- 172.21.15.27:0/4263617434 shutdown_connections 2024-04-16T05:40:49.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.656+0000 7f8105400640 1 --2- 172.21.15.27:0/4263617434 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f80e4069220 0x7f80e406b6d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:49.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.656+0000 7f8105400640 1 --2- 172.21.15.27:0/4263617434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f81001014f0 0x7f810010f2f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:49.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.656+0000 7f8105400640 1 --2- 172.21.15.27:0/4263617434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8100065040 0x7f810010edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:49.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.657+0000 7f8105400640 1 -- 172.21.15.27:0/4263617434 >> 172.21.15.27:0/4263617434 conn(0x7f81000f6680 msgr2=0x7f81000f6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:49.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.657+0000 7f8105400640 1 -- 172.21.15.27:0/4263617434 shutdown_connections 2024-04-16T05:40:49.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:49.657+0000 7f8105400640 1 -- 172.21.15.27:0/4263617434 wait complete. 2024-04-16T05:40:49.673 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 15, 'max_mds': 1, 'flags': 18} 2024-04-16T05:40:49.674 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-16T05:40:49.702 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-16T05:40:49.702 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-04-16T05:40:49.703 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-16T05:40:49.703 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-16T05:40:49.703 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-16T05:40:49.703 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-16T05:40:49.703 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-16T05:40:49.703 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-16T05:40:49.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:40:49.703 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2024-04-16T05:40:49.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:40:49.755 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link delete ceph-brx 2024-04-16T05:40:49.837 INFO:teuthology.orchestra.run.smithi122.stderr:Cannot find device "ceph-brx" 2024-04-16T05:40:49.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T05:40:49.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:40:49.841 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-04-16T05:40:49.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:40:49.883 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2024-04-16T05:40:49.992 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:49 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/3781921552' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T05:40:49.992 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:49 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:40:49.992 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:49 smithi027 ceph-mon[29206]: pgmap v244: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s rd, 0 op/s 2024-04-16T05:40:49.992 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:49 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/4263617434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T05:40:50.005 INFO:teuthology.orchestra.run.smithi027.stderr:Cannot find device "ceph-brx" 2024-04-16T05:40:50.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T05:40:50.009 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-16T05:40:50.009 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph fs ls 2024-04-16T05:40:50.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:49 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/3781921552' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T05:40:50.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:49 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:40:50.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:49 smithi122 ceph-mon[35998]: pgmap v244: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s rd, 0 op/s 2024-04-16T05:40:50.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:49 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/4263617434' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T05:40:50.489 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:40:51.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:51 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:51.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:51 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:51.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:51 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:40:51.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:51 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:40:51.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:51 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:51.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:51 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:51.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:51 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:51.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:51 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:40:51.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:51 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:40:51.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:51 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:40:52.393 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:52 smithi027 ceph-mon[29206]: pgmap v245: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:52.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:52 smithi122 ceph-mon[35998]: pgmap v245: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:52.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.906+0000 7f1d41a00700 1 -- 172.21.15.27:0/2600863493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d3c105820 msgr2=0x7f1d3c0fee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:52.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.906+0000 7f1d41a00700 1 --2- 172.21.15.27:0/2600863493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d3c105820 0x7f1d3c0fee90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1d28004940 tx=0x7f1d28009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:52.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.906+0000 7f1d41a00700 1 -- 172.21.15.27:0/2600863493 shutdown_connections 2024-04-16T05:40:52.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.906+0000 7f1d41a00700 1 --2- 172.21.15.27:0/2600863493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d3c105820 0x7f1d3c0fee90 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:52.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.906+0000 7f1d41a00700 1 --2- 172.21.15.27:0/2600863493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d3c104e80 0x7f1d3c105250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:52.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.906+0000 7f1d41a00700 1 -- 172.21.15.27:0/2600863493 >> 172.21.15.27:0/2600863493 conn(0x7f1d3c0fab10 msgr2=0x7f1d3c0fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:52.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.906+0000 7f1d41a00700 1 -- 172.21.15.27:0/2600863493 shutdown_connections 2024-04-16T05:40:52.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.906+0000 7f1d41a00700 1 -- 172.21.15.27:0/2600863493 wait complete. 2024-04-16T05:40:52.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.907+0000 7f1d41a00700 1 Processor -- start 2024-04-16T05:40:52.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.907+0000 7f1d41a00700 1 -- start start 2024-04-16T05:40:52.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.908+0000 7f1d41a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d3c104e80 0x7f1d3c115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:52.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.908+0000 7f1d41a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d3c105820 0x7f1d3c116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:52.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.908+0000 7f1d41a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c1167a0 con 0x7f1d3c105820 2024-04-16T05:40:52.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.908+0000 7f1d41a00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c10fb90 con 0x7f1d3c104e80 2024-04-16T05:40:52.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.908+0000 7f1d3b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d3c105820 0x7f1d3c116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:52.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.908+0000 7f1d3b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d3c105820 0x7f1d3c116150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58704/0 (socket says 172.21.15.27:58704) 2024-04-16T05:40:52.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.908+0000 7f1d3b400700 1 -- 172.21.15.27:0/2843971216 learned_addr learned my addr 172.21.15.27:0/2843971216 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:40:52.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.908+0000 7f1d3be00700 1 --2- 172.21.15.27:0/2843971216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d3c104e80 0x7f1d3c115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:52.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.909+0000 7f1d3b400700 1 -- 172.21.15.27:0/2843971216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d3c104e80 msgr2=0x7f1d3c115c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:52.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.909+0000 7f1d3b400700 1 --2- 172.21.15.27:0/2843971216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d3c104e80 0x7f1d3c115c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:52.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.909+0000 7f1d3b400700 1 -- 172.21.15.27:0/2843971216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d28005040 con 0x7f1d3c105820 2024-04-16T05:40:52.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.909+0000 7f1d3be00700 1 --2- 172.21.15.27:0/2843971216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d3c104e80 0x7f1d3c115c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T05:40:52.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.909+0000 7f1d3b400700 1 --2- 172.21.15.27:0/2843971216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d3c105820 0x7f1d3c116150 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1d28009100 tx=0x7f1d28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:52.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.909+0000 7f1d33e00700 1 -- 172.21.15.27:0/2843971216 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d2800e040 con 0x7f1d3c105820 2024-04-16T05:40:52.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.909+0000 7f1d33e00700 1 -- 172.21.15.27:0/2843971216 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d28004290 con 0x7f1d3c105820 2024-04-16T05:40:52.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.909+0000 7f1d41a00700 1 -- 172.21.15.27:0/2843971216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d3c10fe10 con 0x7f1d3c105820 2024-04-16T05:40:52.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.909+0000 7f1d33e00700 1 -- 172.21.15.27:0/2843971216 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d2801c3f0 con 0x7f1d3c105820 2024-04-16T05:40:52.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.910+0000 7f1d41a00700 1 -- 172.21.15.27:0/2843971216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d3c1103d0 con 0x7f1d3c105820 2024-04-16T05:40:52.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.911+0000 7f1d41a00700 1 -- 172.21.15.27:0/2843971216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d3c1005d0 con 0x7f1d3c105820 2024-04-16T05:40:52.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.912+0000 7f1d33e00700 1 -- 172.21.15.27:0/2843971216 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f1d28013790 con 0x7f1d3c105820 2024-04-16T05:40:52.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.913+0000 7f1d33e00700 1 --2- 172.21.15.27:0/2843971216 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f1d200692a0 0x7f1d2006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:40:52.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.913+0000 7f1d33e00700 1 -- 172.21.15.27:0/2843971216 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1d28088480 con 0x7f1d3c105820 2024-04-16T05:40:52.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.913+0000 7f1d3be00700 1 --2- 172.21.15.27:0/2843971216 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f1d200692a0 0x7f1d2006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:40:52.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.914+0000 7f1d3be00700 1 --2- 172.21.15.27:0/2843971216 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f1d200692a0 0x7f1d2006b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009fd0 tx=0x7f1d2c009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:40:52.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:52.917+0000 7f1d33e00700 1 -- 172.21.15.27:0/2843971216 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f1d280581d0 con 0x7f1d3c105820 2024-04-16T05:40:53.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:53.185+0000 7f1d41a00700 1 -- 172.21.15.27:0/2843971216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1d3c10bf60 con 0x7f1d3c105820 2024-04-16T05:40:53.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:53.186+0000 7f1d33e00700 1 -- 172.21.15.27:0/2843971216 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+83 (secure 0 0 0) 0x7f1d28013420 con 0x7f1d3c105820 2024-04-16T05:40:53.185 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-16T05:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:53.189+0000 7f1d41a00700 1 -- 172.21.15.27:0/2843971216 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f1d200692a0 msgr2=0x7f1d2006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:53.189+0000 7f1d41a00700 1 --2- 172.21.15.27:0/2843971216 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f1d200692a0 0x7f1d2006b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009fd0 tx=0x7f1d2c009ee0 comp rx=0 tx=0).stop 2024-04-16T05:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:53.189+0000 7f1d41a00700 1 -- 172.21.15.27:0/2843971216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d3c105820 msgr2=0x7f1d3c116150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:53.189+0000 7f1d41a00700 1 --2- 172.21.15.27:0/2843971216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d3c105820 0x7f1d3c116150 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1d28009100 tx=0x7f1d28009e30 comp rx=0 tx=0).stop 2024-04-16T05:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:53.190+0000 7f1d41a00700 1 -- 172.21.15.27:0/2843971216 shutdown_connections 2024-04-16T05:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:53.190+0000 7f1d41a00700 1 --2- 172.21.15.27:0/2843971216 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f1d200692a0 0x7f1d2006b750 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:53.190+0000 7f1d41a00700 1 --2- 172.21.15.27:0/2843971216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d3c104e80 0x7f1d3c115c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:53.190+0000 7f1d41a00700 1 --2- 172.21.15.27:0/2843971216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d3c105820 0x7f1d3c116150 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:53.190+0000 7f1d41a00700 1 -- 172.21.15.27:0/2843971216 >> 172.21.15.27:0/2843971216 conn(0x7f1d3c0fab10 msgr2=0x7f1d3c0fcb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:40:53.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:53.190+0000 7f1d41a00700 1 -- 172.21.15.27:0/2843971216 shutdown_connections 2024-04-16T05:40:53.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:53.190+0000 7f1d41a00700 1 -- 172.21.15.27:0/2843971216 wait complete. 2024-04-16T05:40:53.458 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:53 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/2843971216' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-16T05:40:53.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:53 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/2843971216' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-16T05:40:54.104 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-16T05:40:54.104 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T05:40:54.104 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2024-04-16T05:40:54.104 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-16T05:40:54.105 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-16T05:40:54.105 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-16T05:40:54.105 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-16T05:40:54.105 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T05:40:54.105 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-16T05:40:54.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:40:54.105 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2024-04-16T05:40:54.168 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T05:40:54.168 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T05:40:54.168 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T05:40:54.168 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-04-16T05:40:54.169 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2024-04-16T05:40:54.169 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-04-16T05:40:54.169 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T05:40:54.169 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2024-04-16T05:40:54.169 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2024-04-16T05:40:54.169 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T05:40:54.169 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2024-04-16T05:40:54.169 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2024-04-16T05:40:54.169 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T05:40:54.170 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2024-04-16T05:40:54.170 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2024-04-16T05:40:54.170 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-16T05:40:54.170 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 41967sec preferred_lft 41967sec 2024-04-16T05:40:54.170 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link noprefixroute 2024-04-16T05:40:54.170 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2024-04-16T05:40:54.170 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T05:40:54.170 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2024-04-16T05:40:54.170 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2024-04-16T05:40:54.171 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-16T05:40:54.171 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T05:40:54.171 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-16T05:40:54.171 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add name ceph-brx type bridge 2024-04-16T05:40:54.171 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr flush dev ceph-brx 2024-04-16T05:40:54.171 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set ceph-brx up 2024-04-16T05:40:54.172 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-16T05:40:54.172 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-16T05:40:54.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T05:40:54.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:54 smithi027 ceph-mon[29206]: pgmap v246: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T05:40:54.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:40:54.762 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T05:40:54.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:54 smithi122 ceph-mon[35998]: pgmap v246: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:54.850 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-04-16T05:40:54.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:40:54.855 DEBUG:teuthology.orchestra.run.smithi027:> route 2024-04-16T05:40:54.910 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2024-04-16T05:40:54.910 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T05:40:54.910 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-16T05:40:54.910 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-16T05:40:54.910 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T05:40:54.911 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T05:40:54.911 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-16T05:40:54.911 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-16T05:40:54.911 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-16T05:40:54.911 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-16T05:40:54.911 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-16T05:40:55.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T05:40:55.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T05:40:55.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:40:55.774 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2024-04-16T05:40:55.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:40:55.836 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list-id 2024-04-16T05:40:55.891 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T05:40:55.891 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-16T05:40:55.891 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T05:40:55.891 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-16T05:40:55.891 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-16T05:40:56.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T05:40:56.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T05:40:56.129 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-16T05:40:56.129 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T05:40:56.130 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-16T05:40:56.130 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-16T05:40:56.130 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-16T05:40:56.130 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-16T05:40:56.130 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-16T05:40:56.130 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-16T05:40:56.130 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-16T05:40:56.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T05:40:56.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T05:40:56.587 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T05:40:56.587 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2024-04-16T05:40:56.587 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2024-04-16T05:40:56.588 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set dev brx.0 master ceph-brx 2024-04-16T05:40:56.588 DEBUG:teuthology.orchestra.run.smithi027:> ') 2024-04-16T05:40:56.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T05:40:56.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:56 smithi027 ceph-mon[29206]: pgmap v247: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:56.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T05:40:56.843 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-16T05:40:56.843 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T05:40:56.843 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-16T05:40:56.890 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-16T05:40:56.890 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T05:40:56.890 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-16T05:40:56.984 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-04-16T05:40:57.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:56 smithi122 ceph-mon[35998]: pgmap v247: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:57.093 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T05:40:57.149 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-04-16T05:40:57.149 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-04-16T05:40:57.149 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-04-16T05:40:57.149 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-04-16T05:40:57.150 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-04-16T05:40:57.150 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-04-16T05:40:57.150 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-04-16T05:40:57.150 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-04-16T05:40:57.150 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-04-16T05:40:57.150 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-04-16T05:40:57.150 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-04-16T05:40:57.150 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-04-16T05:40:57.150 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T05:40:57.150 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-04-16T05:40:57.150 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-04-16T05:40:57.150 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-04-16T05:40:57.150 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-04-16T05:40:57.150 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-04-16T05:40:57.150 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T05:40:57.151 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-04-16T05:40:57.151 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T05:40:57.151 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T05:40:57.151 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T05:40:57.151 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-04-16T05:40:57.151 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-04-16T05:40:57.151 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay 2024-04-16T05:40:57.151 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/6469aaea3cab6b38cd1b9b3852dad21f89a05805b33507eda8ed770343fdc6c0/merged 2024-04-16T05:40:57.151 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/f29b51083c197992b63d413e9648eebf7a973a5b76539f4b6d2351cf18a70a77/merged 2024-04-16T05:40:57.151 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/0 2024-04-16T05:40:57.151 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T05:40:57.151 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/495d4c18fc420e7d459aabb6c90fa04e2138954dc851a181b83178c342441c49/merged 2024-04-16T05:40:57.151 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/58715b3bfee29aa1d59f35d659f244c34845e012fbc5cf682ab710d2772e197b/merged 2024-04-16T05:40:57.151 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/47ba8bc77c3f7aafa4dfc672d9a8a2d91f1bb4ba44a7db154aa5c44e83c28098/merged 2024-04-16T05:40:57.151 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/209734cc168ef0c790bca266e5bef8486d770939885d2cfeaa4a735262fa85ab/merged 2024-04-16T05:40:57.152 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/b59cc37dcb6f962103c57ca181a19b8a918d2a245ea668727db730159c1c4e5c/merged 2024-04-16T05:40:57.152 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/ec18f971e1ea72884a3c0f8e97dddf64c8de6b8d522178786811472b30cd8e05/merged 2024-04-16T05:40:57.152 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/7e1c07779823a7a62d6f34d20f9abb4505ae4e09155c2c2a0a40b459bffd5008/merged 2024-04-16T05:40:57.152 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/c56d2e061bd7746929bc861f397b68bf1a64caf34953dc1f8fbfbed22a98d51d/merged 2024-04-16T05:40:57.152 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/8b135047d2daa0dcb467db71e743332b425cfb5bcc2e323fc60fffad2a413637/merged 2024-04-16T05:40:57.152 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/ca0dc1e90b188ecdb83c6367ff8a37e3b19eda4d794dcd63bfe9aa186f417fb3/merged 2024-04-16T05:40:57.152 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-04-16T05:40:57.152 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T05:40:57.152 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T05:40:57.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:40:57.153 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-04-16T05:40:57.206 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-16T05:40:57.206 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-16T05:40:57.211 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-04-16T05:40:57.299 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T05:40:57.351 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-04-16T05:40:57.351 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-04-16T05:40:57.351 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-04-16T05:40:57.351 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-04-16T05:40:57.351 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-04-16T05:40:57.351 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-04-16T05:40:57.351 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-04-16T05:40:57.351 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-04-16T05:40:57.351 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-04-16T05:40:57.351 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-04-16T05:40:57.351 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-04-16T05:40:57.351 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-04-16T05:40:57.352 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T05:40:57.352 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-04-16T05:40:57.352 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-04-16T05:40:57.352 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-04-16T05:40:57.352 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-04-16T05:40:57.352 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-04-16T05:40:57.352 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T05:40:57.352 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-04-16T05:40:57.352 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T05:40:57.352 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T05:40:57.352 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T05:40:57.352 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-04-16T05:40:57.352 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-04-16T05:40:57.352 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay 2024-04-16T05:40:57.352 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/6469aaea3cab6b38cd1b9b3852dad21f89a05805b33507eda8ed770343fdc6c0/merged 2024-04-16T05:40:57.353 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/f29b51083c197992b63d413e9648eebf7a973a5b76539f4b6d2351cf18a70a77/merged 2024-04-16T05:40:57.353 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/0 2024-04-16T05:40:57.353 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T05:40:57.353 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/495d4c18fc420e7d459aabb6c90fa04e2138954dc851a181b83178c342441c49/merged 2024-04-16T05:40:57.353 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/58715b3bfee29aa1d59f35d659f244c34845e012fbc5cf682ab710d2772e197b/merged 2024-04-16T05:40:57.353 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/47ba8bc77c3f7aafa4dfc672d9a8a2d91f1bb4ba44a7db154aa5c44e83c28098/merged 2024-04-16T05:40:57.353 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/209734cc168ef0c790bca266e5bef8486d770939885d2cfeaa4a735262fa85ab/merged 2024-04-16T05:40:57.353 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/b59cc37dcb6f962103c57ca181a19b8a918d2a245ea668727db730159c1c4e5c/merged 2024-04-16T05:40:57.353 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/ec18f971e1ea72884a3c0f8e97dddf64c8de6b8d522178786811472b30cd8e05/merged 2024-04-16T05:40:57.353 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/7e1c07779823a7a62d6f34d20f9abb4505ae4e09155c2c2a0a40b459bffd5008/merged 2024-04-16T05:40:57.353 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/c56d2e061bd7746929bc861f397b68bf1a64caf34953dc1f8fbfbed22a98d51d/merged 2024-04-16T05:40:57.353 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/8b135047d2daa0dcb467db71e743332b425cfb5bcc2e323fc60fffad2a413637/merged 2024-04-16T05:40:57.353 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/ca0dc1e90b188ecdb83c6367ff8a37e3b19eda4d794dcd63bfe9aa186f417fb3/merged 2024-04-16T05:40:57.353 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-04-16T05:40:57.353 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T05:40:57.354 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T05:40:57.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:40:57.354 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-04-16T05:40:57.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-04-16T05:40:57.452+0000 7f6d76acf300 -1 init, newargv = 0x564fc572ceb0 newargc=16 2024-04-16T05:40:57.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[76806]: starting ceph client 2024-04-16T05:40:57.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:FUSE library version: 2.9.9 2024-04-16T05:40:57.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[76806]: starting fuse 2024-04-16T05:40:57.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stdout:ceph_fuse: starting fuse loop 2024-04-16T05:40:57.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 2, opcode: INIT (26), nodeid: 0, insize: 104, pid: 0 2024-04-16T05:40:57.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:INIT: 7.40 2024-04-16T05:40:57.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:flags=0x73fffffb 2024-04-16T05:40:57.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:max_readahead=0x00020000 2024-04-16T05:40:57.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: INIT: 7.19 2024-04-16T05:40:57.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: flags=0x0000043b 2024-04-16T05:40:57.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: max_readahead=0x00020000 2024-04-16T05:40:57.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: max_write=0x00020000 2024-04-16T05:40:57.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: max_background=0 2024-04-16T05:40:57.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: congestion_threshold=0 2024-04-16T05:40:57.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 2, success, outsize: 40 2024-04-16T05:40:57.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:2024-04-16T05:40:57.465+0000 7f6d76acf300 -1 multithreaded fuse 2024-04-16T05:40:57.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 4, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 76834 2024-04-16T05:40:57.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 4, success, outsize: 120 2024-04-16T05:40:58.499 DEBUG:teuthology.orchestra.run.smithi027:> sudo modprobe fuse 2024-04-16T05:40:58.618 DEBUG:teuthology.orchestra.run.smithi027:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T05:40:58.681 INFO:teuthology.orchestra.run.smithi027.stdout:/proc 2024-04-16T05:40:58.681 INFO:teuthology.orchestra.run.smithi027.stdout:/sys 2024-04-16T05:40:58.681 INFO:teuthology.orchestra.run.smithi027.stdout:/dev 2024-04-16T05:40:58.681 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/security 2024-04-16T05:40:58.681 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/shm 2024-04-16T05:40:58.681 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/pts 2024-04-16T05:40:58.681 INFO:teuthology.orchestra.run.smithi027.stdout:/run 2024-04-16T05:40:58.682 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/cgroup 2024-04-16T05:40:58.682 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/pstore 2024-04-16T05:40:58.682 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/bpf 2024-04-16T05:40:58.682 INFO:teuthology.orchestra.run.smithi027.stdout:/ 2024-04-16T05:40:58.682 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/selinux 2024-04-16T05:40:58.682 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T05:40:58.682 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/mqueue 2024-04-16T05:40:58.682 INFO:teuthology.orchestra.run.smithi027.stdout:/dev/hugepages 2024-04-16T05:40:58.682 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/debug 2024-04-16T05:40:58.682 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/tracing 2024-04-16T05:40:58.682 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/fs/fuse/connections 2024-04-16T05:40:58.682 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T05:40:58.682 INFO:teuthology.orchestra.run.smithi027.stdout:/sys/kernel/config 2024-04-16T05:40:58.682 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T05:40:58.682 INFO:teuthology.orchestra.run.smithi027.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T05:40:58.683 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T05:40:58.683 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/1000 2024-04-16T05:40:58.683 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/ceph 2024-04-16T05:40:58.683 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay 2024-04-16T05:40:58.683 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/6469aaea3cab6b38cd1b9b3852dad21f89a05805b33507eda8ed770343fdc6c0/merged 2024-04-16T05:40:58.683 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/f29b51083c197992b63d413e9648eebf7a973a5b76539f4b6d2351cf18a70a77/merged 2024-04-16T05:40:58.683 INFO:teuthology.orchestra.run.smithi027.stdout:/run/user/0 2024-04-16T05:40:58.683 INFO:teuthology.orchestra.run.smithi027.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T05:40:58.683 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/495d4c18fc420e7d459aabb6c90fa04e2138954dc851a181b83178c342441c49/merged 2024-04-16T05:40:58.683 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/58715b3bfee29aa1d59f35d659f244c34845e012fbc5cf682ab710d2772e197b/merged 2024-04-16T05:40:58.683 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/47ba8bc77c3f7aafa4dfc672d9a8a2d91f1bb4ba44a7db154aa5c44e83c28098/merged 2024-04-16T05:40:58.683 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/209734cc168ef0c790bca266e5bef8486d770939885d2cfeaa4a735262fa85ab/merged 2024-04-16T05:40:58.683 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/b59cc37dcb6f962103c57ca181a19b8a918d2a245ea668727db730159c1c4e5c/merged 2024-04-16T05:40:58.683 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/ec18f971e1ea72884a3c0f8e97dddf64c8de6b8d522178786811472b30cd8e05/merged 2024-04-16T05:40:58.683 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/7e1c07779823a7a62d6f34d20f9abb4505ae4e09155c2c2a0a40b459bffd5008/merged 2024-04-16T05:40:58.684 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/c56d2e061bd7746929bc861f397b68bf1a64caf34953dc1f8fbfbed22a98d51d/merged 2024-04-16T05:40:58.684 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/8b135047d2daa0dcb467db71e743332b425cfb5bcc2e323fc60fffad2a413637/merged 2024-04-16T05:40:58.684 INFO:teuthology.orchestra.run.smithi027.stdout:/var/lib/containers/storage/overlay/ca0dc1e90b188ecdb83c6367ff8a37e3b19eda4d794dcd63bfe9aa186f417fb3/merged 2024-04-16T05:40:58.684 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns 2024-04-16T05:40:58.684 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T05:40:58.684 INFO:teuthology.orchestra.run.smithi027.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-16T05:40:58.684 INFO:teuthology.orchestra.run.smithi027.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-16T05:40:58.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:40:58.685 DEBUG:teuthology.orchestra.run.smithi027:> ls /sys/fs/fuse/connections 2024-04-16T05:40:58.740 INFO:teuthology.orchestra.run.smithi027.stdout:148 2024-04-16T05:40:58.740 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [148] 2024-04-16T05:40:58.741 DEBUG:teuthology.orchestra.run.smithi027:> sudo stdin-killer -- python3 -c ' 2024-04-16T05:40:58.741 DEBUG:teuthology.orchestra.run.smithi027:> import glob 2024-04-16T05:40:58.741 DEBUG:teuthology.orchestra.run.smithi027:> import re 2024-04-16T05:40:58.741 DEBUG:teuthology.orchestra.run.smithi027:> import os 2024-04-16T05:40:58.741 DEBUG:teuthology.orchestra.run.smithi027:> import subprocess 2024-04-16T05:40:58.741 DEBUG:teuthology.orchestra.run.smithi027:> 2024-04-16T05:40:58.741 DEBUG:teuthology.orchestra.run.smithi027:> def _find_admin_socket(client_name): 2024-04-16T05:40:58.741 DEBUG:teuthology.orchestra.run.smithi027:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-16T05:40:58.741 DEBUG:teuthology.orchestra.run.smithi027:> files = glob.glob(asok_path) 2024-04-16T05:40:58.741 DEBUG:teuthology.orchestra.run.smithi027:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-16T05:40:58.741 DEBUG:teuthology.orchestra.run.smithi027:> 2024-04-16T05:40:58.741 DEBUG:teuthology.orchestra.run.smithi027:> # Given a non-glob path, it better be there 2024-04-16T05:40:58.741 DEBUG:teuthology.orchestra.run.smithi027:> if "*" not in asok_path: 2024-04-16T05:40:58.741 DEBUG:teuthology.orchestra.run.smithi027:> assert(len(files) == 1) 2024-04-16T05:40:58.741 DEBUG:teuthology.orchestra.run.smithi027:> return files[0] 2024-04-16T05:40:58.741 DEBUG:teuthology.orchestra.run.smithi027:> 2024-04-16T05:40:58.742 DEBUG:teuthology.orchestra.run.smithi027:> for f in files: 2024-04-16T05:40:58.742 DEBUG:teuthology.orchestra.run.smithi027:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-16T05:40:58.742 DEBUG:teuthology.orchestra.run.smithi027:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-16T05:40:58.742 DEBUG:teuthology.orchestra.run.smithi027:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-16T05:40:58.742 DEBUG:teuthology.orchestra.run.smithi027:> contents = proc_f.read() 2024-04-16T05:40:58.742 DEBUG:teuthology.orchestra.run.smithi027:> if mountpoint in contents: 2024-04-16T05:40:58.742 DEBUG:teuthology.orchestra.run.smithi027:> return f 2024-04-16T05:40:58.742 DEBUG:teuthology.orchestra.run.smithi027:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-16T05:40:58.742 DEBUG:teuthology.orchestra.run.smithi027:> 2024-04-16T05:40:58.742 DEBUG:teuthology.orchestra.run.smithi027:> print(_find_admin_socket("client.0")) 2024-04-16T05:40:58.742 DEBUG:teuthology.orchestra.run.smithi027:> ' 2024-04-16T05:40:58.936 INFO:teuthology.orchestra.run.smithi027.stdout:/var/run/ceph/ceph-client.0.76806.asok 2024-04-16T05:40:58.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:40:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T05:40:58.951 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:40:58 smithi027 ceph-mon[29206]: pgmap v248: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:58.955 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.76806.asok 2024-04-16T05:40:58.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:40:58.956 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.76806.asok status 2024-04-16T05:40:59.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:40:58 smithi122 ceph-mon[35998]: pgmap v248: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:40:59.206 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:40:59.206 INFO:teuthology.orchestra.run.smithi027.stdout: "metadata": { 2024-04-16T05:40:59.206 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_sha1": "45317f95c5587af98fd8eaefff9013a4ba51d2f7", 2024-04-16T05:40:59.206 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph_version": "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)", 2024-04-16T05:40:59.206 INFO:teuthology.orchestra.run.smithi027.stdout: "entity_id": "0", 2024-04-16T05:40:59.206 INFO:teuthology.orchestra.run.smithi027.stdout: "hostname": "smithi027", 2024-04-16T05:40:59.207 INFO:teuthology.orchestra.run.smithi027.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-16T05:40:59.207 INFO:teuthology.orchestra.run.smithi027.stdout: "pid": "76806", 2024-04-16T05:40:59.207 INFO:teuthology.orchestra.run.smithi027.stdout: "root": "/" 2024-04-16T05:40:59.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:40:59.207 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_count": 0, 2024-04-16T05:40:59.207 INFO:teuthology.orchestra.run.smithi027.stdout: "dentry_pinned_count": 0, 2024-04-16T05:40:59.207 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 14518, 2024-04-16T05:40:59.207 INFO:teuthology.orchestra.run.smithi027.stdout: "inst": { 2024-04-16T05:40:59.207 INFO:teuthology.orchestra.run.smithi027.stdout: "name": { 2024-04-16T05:40:59.207 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "client", 2024-04-16T05:40:59.207 INFO:teuthology.orchestra.run.smithi027.stdout: "num": 14518 2024-04-16T05:40:59.207 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:40:59.207 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-04-16T05:40:59.207 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-04-16T05:40:59.207 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-04-16T05:40:59.208 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 676551963 2024-04-16T05:40:59.208 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:40:59.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:40:59.208 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": { 2024-04-16T05:40:59.208 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-04-16T05:40:59.208 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "192.168.0.1:0", 2024-04-16T05:40:59.208 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 676551963 2024-04-16T05:40:59.208 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:40:59.208 INFO:teuthology.orchestra.run.smithi027.stdout: "inst_str": "client.14518 192.168.0.1:0/676551963", 2024-04-16T05:40:59.208 INFO:teuthology.orchestra.run.smithi027.stdout: "addr_str": "192.168.0.1:0/676551963", 2024-04-16T05:40:59.208 INFO:teuthology.orchestra.run.smithi027.stdout: "inode_count": 1, 2024-04-16T05:40:59.208 INFO:teuthology.orchestra.run.smithi027.stdout: "mds_epoch": 15, 2024-04-16T05:40:59.208 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch": 46, 2024-04-16T05:40:59.208 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_epoch_barrier": 0, 2024-04-16T05:40:59.208 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklisted": false, 2024-04-16T05:40:59.209 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs" 2024-04-16T05:40:59.209 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:40:59.222 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- ceph fs ls 2024-04-16T05:40:59.753 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:41:00.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:00 smithi027 ceph-mon[29206]: pgmap v249: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T05:41:01.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:00 smithi122 ceph-mon[35998]: pgmap v249: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T05:41:02.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.229+0000 7f1735800700 1 -- 172.21.15.27:0/1258115846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17300656b0 msgr2=0x7f1730065b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:02.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.229+0000 7f1735800700 1 --2- 172.21.15.27:0/1258115846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17300656b0 0x7f1730065b20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1718004940 tx=0x7f1718009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:02.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.229+0000 7f1735800700 1 -- 172.21.15.27:0/1258115846 shutdown_connections 2024-04-16T05:41:02.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.229+0000 7f1735800700 1 --2- 172.21.15.27:0/1258115846 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17300656b0 0x7f1730065b20 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:02.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.229+0000 7f1735800700 1 --2- 172.21.15.27:0/1258115846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1730102e60 0x7f1730103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:02.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.229+0000 7f1735800700 1 -- 172.21.15.27:0/1258115846 >> 172.21.15.27:0/1258115846 conn(0x7f1730072bc0 msgr2=0x7f1730072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:02.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.229+0000 7f1735800700 1 -- 172.21.15.27:0/1258115846 shutdown_connections 2024-04-16T05:41:02.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.229+0000 7f1735800700 1 -- 172.21.15.27:0/1258115846 wait complete. 2024-04-16T05:41:02.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.230+0000 7f1735800700 1 Processor -- start 2024-04-16T05:41:02.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.230+0000 7f1735800700 1 -- start start 2024-04-16T05:41:02.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.231+0000 7f1735800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17300656b0 0x7f1730115d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:02.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.231+0000 7f1735800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1730102e60 0x7f1730116240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:02.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.231+0000 7f1735800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1730116890 con 0x7f17300656b0 2024-04-16T05:41:02.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.231+0000 7f1735800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173010fc60 con 0x7f1730102e60 2024-04-16T05:41:02.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.231+0000 7f172ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17300656b0 0x7f1730115d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:02.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.231+0000 7f172ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17300656b0 0x7f1730115d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41234/0 (socket says 172.21.15.27:41234) 2024-04-16T05:41:02.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.231+0000 7f172ea00700 1 -- 172.21.15.27:0/3052198549 learned_addr learned my addr 172.21.15.27:0/3052198549 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:41:02.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.231+0000 7f1726000700 1 --2- 172.21.15.27:0/3052198549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1730102e60 0x7f1730116240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:02.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.231+0000 7f172ea00700 1 -- 172.21.15.27:0/3052198549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1730102e60 msgr2=0x7f1730116240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:02.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.231+0000 7f172ea00700 1 --2- 172.21.15.27:0/3052198549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1730102e60 0x7f1730116240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:02.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.231+0000 7f172ea00700 1 -- 172.21.15.27:0/3052198549 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1718005040 con 0x7f17300656b0 2024-04-16T05:41:02.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.232+0000 7f1726000700 1 --2- 172.21.15.27:0/3052198549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1730102e60 0x7f1730116240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:41:02.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.232+0000 7f172ea00700 1 --2- 172.21.15.27:0/3052198549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17300656b0 0x7f1730115d00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1720002a50 tx=0x7f17200046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:02.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.232+0000 7f1727400700 1 -- 172.21.15.27:0/3052198549 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1720007cc0 con 0x7f17300656b0 2024-04-16T05:41:02.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.232+0000 7f1727400700 1 -- 172.21.15.27:0/3052198549 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f172000f040 con 0x7f17300656b0 2024-04-16T05:41:02.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.232+0000 7f1727400700 1 -- 172.21.15.27:0/3052198549 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1720009640 con 0x7f17300656b0 2024-04-16T05:41:02.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.232+0000 7f1735800700 1 -- 172.21.15.27:0/3052198549 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f173010ff40 con 0x7f17300656b0 2024-04-16T05:41:02.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.232+0000 7f1735800700 1 -- 172.21.15.27:0/3052198549 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17301104b0 con 0x7f17300656b0 2024-04-16T05:41:02.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.234+0000 7f1735800700 1 -- 172.21.15.27:0/3052198549 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17301069d0 con 0x7f17300656b0 2024-04-16T05:41:02.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.235+0000 7f1727400700 1 -- 172.21.15.27:0/3052198549 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f17200097c0 con 0x7f17300656b0 2024-04-16T05:41:02.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.236+0000 7f1727400700 1 --2- 172.21.15.27:0/3052198549 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f17140692b0 0x7f171406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:02.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.236+0000 7f1727400700 1 -- 172.21.15.27:0/3052198549 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1720087000 con 0x7f17300656b0 2024-04-16T05:41:02.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.236+0000 7f1726000700 1 --2- 172.21.15.27:0/3052198549 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f17140692b0 0x7f171406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:02.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.237+0000 7f1726000700 1 --2- 172.21.15.27:0/3052198549 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f17140692b0 0x7f171406b760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1718004f90 tx=0x7f1718009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:02.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.241+0000 7f1727400700 1 -- 172.21.15.27:0/3052198549 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f1720056cf0 con 0x7f17300656b0 2024-04-16T05:41:02.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.513+0000 7f1735800700 1 -- 172.21.15.27:0/3052198549 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f173004bed0 con 0x7f17300656b0 2024-04-16T05:41:02.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.514+0000 7f1727400700 1 -- 172.21.15.27:0/3052198549 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+83 (secure 0 0 0) 0x7f1720056880 con 0x7f17300656b0 2024-04-16T05:41:02.513 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-04-16T05:41:02.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.517+0000 7f1735800700 1 -- 172.21.15.27:0/3052198549 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f17140692b0 msgr2=0x7f171406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:02.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.517+0000 7f1735800700 1 --2- 172.21.15.27:0/3052198549 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f17140692b0 0x7f171406b760 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1718004f90 tx=0x7f1718009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:02.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.517+0000 7f1735800700 1 -- 172.21.15.27:0/3052198549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17300656b0 msgr2=0x7f1730115d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:02.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.517+0000 7f1735800700 1 --2- 172.21.15.27:0/3052198549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17300656b0 0x7f1730115d00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1720002a50 tx=0x7f17200046b0 comp rx=0 tx=0).stop 2024-04-16T05:41:02.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.517+0000 7f1735800700 1 -- 172.21.15.27:0/3052198549 shutdown_connections 2024-04-16T05:41:02.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.517+0000 7f1735800700 1 --2- 172.21.15.27:0/3052198549 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f17140692b0 0x7f171406b760 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:02.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.517+0000 7f1735800700 1 --2- 172.21.15.27:0/3052198549 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f17300656b0 0x7f1730115d00 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:02.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.517+0000 7f1735800700 1 --2- 172.21.15.27:0/3052198549 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1730102e60 0x7f1730116240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:02.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.517+0000 7f1735800700 1 -- 172.21.15.27:0/3052198549 >> 172.21.15.27:0/3052198549 conn(0x7f1730072bc0 msgr2=0x7f17300facc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:02.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.517+0000 7f1735800700 1 -- 172.21.15.27:0/3052198549 shutdown_connections 2024-04-16T05:41:02.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:02.518+0000 7f1735800700 1 -- 172.21.15.27:0/3052198549 wait complete. 2024-04-16T05:41:02.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:02 smithi027 ceph-mon[29206]: pgmap v250: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T05:41:02.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:02 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/3052198549' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-16T05:41:03.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:02 smithi122 ceph-mon[35998]: pgmap v250: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T05:41:03.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:02 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/3052198549' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-16T05:41:03.375 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-16T05:41:03.376 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-16T05:41:03.376 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi122.front.sepia.ceph.com 2024-04-16T05:41:03.376 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-16T05:41:03.376 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-16T05:41:03.376 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-16T05:41:03.376 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-16T05:41:03.376 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-16T05:41:03.376 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-16T05:41:03.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:41:03.377 DEBUG:teuthology.orchestra.run.smithi122:> ip addr 2024-04-16T05:41:03.435 INFO:teuthology.orchestra.run.smithi122.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-16T05:41:03.435 INFO:teuthology.orchestra.run.smithi122.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-16T05:41:03.436 INFO:teuthology.orchestra.run.smithi122.stdout: inet 127.0.0.1/8 scope host lo 2024-04-16T05:41:03.436 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-04-16T05:41:03.436 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 ::1/128 scope host 2024-04-16T05:41:03.436 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-04-16T05:41:03.436 INFO:teuthology.orchestra.run.smithi122.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T05:41:03.436 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e4 brd ff:ff:ff:ff:ff:ff 2024-04-16T05:41:03.436 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f0 2024-04-16T05:41:03.436 INFO:teuthology.orchestra.run.smithi122.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T05:41:03.436 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:d8:d9:e5 brd ff:ff:ff:ff:ff:ff 2024-04-16T05:41:03.436 INFO:teuthology.orchestra.run.smithi122.stdout: altname enp7s0f1 2024-04-16T05:41:03.436 INFO:teuthology.orchestra.run.smithi122.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-16T05:41:03.436 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:14 brd ff:ff:ff:ff:ff:ff 2024-04-16T05:41:03.436 INFO:teuthology.orchestra.run.smithi122.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-16T05:41:03.436 INFO:teuthology.orchestra.run.smithi122.stdout: link/ether 0c:c4:7a:8f:d5:15 brd ff:ff:ff:ff:ff:ff 2024-04-16T05:41:03.437 INFO:teuthology.orchestra.run.smithi122.stdout: inet 172.21.15.122/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-16T05:41:03.437 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft 41946sec preferred_lft 41946sec 2024-04-16T05:41:03.437 INFO:teuthology.orchestra.run.smithi122.stdout: inet6 fe80::ec4:7aff:fe8f:d515/64 scope link 2024-04-16T05:41:03.437 INFO:teuthology.orchestra.run.smithi122.stdout: valid_lft forever preferred_lft forever 2024-04-16T05:41:03.437 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-16T05:41:03.437 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T05:41:03.437 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-04-16T05:41:03.437 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link add name ceph-brx type bridge 2024-04-16T05:41:03.437 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip addr flush dev ceph-brx 2024-04-16T05:41:03.438 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set ceph-brx up 2024-04-16T05:41:03.438 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-16T05:41:03.438 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-04-16T05:41:03.540 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:41:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T05:41:03.949 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:41:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T05:41:03.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:41:03.958 DEBUG:teuthology.orchestra.run.smithi122:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-16T05:41:04.033 INFO:teuthology.orchestra.run.smithi122.stdout:1 2024-04-16T05:41:04.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:41:04.037 DEBUG:teuthology.orchestra.run.smithi122:> route 2024-04-16T05:41:04.093 INFO:teuthology.orchestra.run.smithi122.stdout:Kernel IP routing table 2024-04-16T05:41:04.093 INFO:teuthology.orchestra.run.smithi122.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-16T05:41:04.094 INFO:teuthology.orchestra.run.smithi122.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-16T05:41:04.094 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-16T05:41:04.094 INFO:teuthology.orchestra.run.smithi122.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-16T05:41:04.094 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T05:41:04.094 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-04-16T05:41:04.094 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-16T05:41:04.094 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-16T05:41:04.094 DEBUG:teuthology.orchestra.run.smithi122:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-16T05:41:04.094 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-04-16T05:41:04.191 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:41:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T05:41:04.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:04 smithi122 ceph-mon[35998]: pgmap v251: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T05:41:04.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:04 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:41:04.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:41:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T05:41:04.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:41:04.867 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list 2024-04-16T05:41:04.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:41:04.914 DEBUG:teuthology.orchestra.run.smithi122:> ip netns list-id 2024-04-16T05:41:05.003 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T05:41:05.003 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-04-16T05:41:05.004 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T05:41:05.004 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-16T05:41:05.004 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-04-16T05:41:05.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:04 smithi027 ceph-mon[29206]: pgmap v251: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T05:41:05.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:04 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:41:05.095 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:41:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T05:41:05.178 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:41:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T05:41:05.187 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-16T05:41:05.188 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T05:41:05.188 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-04-16T05:41:05.188 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-16T05:41:05.188 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-16T05:41:05.188 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-16T05:41:05.188 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-16T05:41:05.188 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-16T05:41:05.188 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-04-16T05:41:05.275 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:41:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T05:41:05.507 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:41:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T05:41:05.516 DEBUG:teuthology.orchestra.run.smithi122:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-16T05:41:05.516 DEBUG:teuthology.orchestra.run.smithi122:> set -e 2024-04-16T05:41:05.516 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set brx.0 up 2024-04-16T05:41:05.516 DEBUG:teuthology.orchestra.run.smithi122:> sudo ip link set dev brx.0 master ceph-brx 2024-04-16T05:41:05.516 DEBUG:teuthology.orchestra.run.smithi122:> ') 2024-04-16T05:41:05.603 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:41:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-16T05:41:05.717 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:41:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T05:41:05.726 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-16T05:41:05.726 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T05:41:05.726 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-16T05:41:05.772 INFO:teuthology.orchestra.run.smithi122.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-16T05:41:05.772 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-16T05:41:05.772 DEBUG:teuthology.orchestra.run.smithi122:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-16T05:41:05.862 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-04-16T05:41:05.944 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T05:41:05.994 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-04-16T05:41:05.994 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-04-16T05:41:05.994 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-04-16T05:41:05.994 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-04-16T05:41:05.994 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-04-16T05:41:05.994 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-04-16T05:41:05.994 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-04-16T05:41:05.994 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-04-16T05:41:05.995 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-04-16T05:41:05.995 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-04-16T05:41:05.995 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-04-16T05:41:05.995 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-04-16T05:41:05.995 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T05:41:05.995 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-04-16T05:41:05.995 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-04-16T05:41:05.995 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-04-16T05:41:05.995 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-04-16T05:41:05.995 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-04-16T05:41:05.995 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T05:41:05.995 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T05:41:05.995 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-04-16T05:41:05.995 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T05:41:05.996 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T05:41:05.996 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-04-16T05:41:05.996 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-04-16T05:41:05.996 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/0 2024-04-16T05:41:05.996 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T05:41:05.996 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay 2024-04-16T05:41:05.996 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/4deadc44deea52cd2034db77cfbb4d78387055e3f20ec6eedbbc750a4ba6e6e4/merged 2024-04-16T05:41:05.996 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/619822cbb0be564e77872d1a5b2246f9c86822ea3b6de049a5ae3cbdd30954b7/merged 2024-04-16T05:41:05.996 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/a85cb7c54677585c51ef3b81321e3afc02b05e9a39337ba3f5b436c4774aae8f/merged 2024-04-16T05:41:05.996 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/104c989e7579aaadd5b157ea98e406e7cb88fbe74511b7dc00d369646e95c3f8/merged 2024-04-16T05:41:05.996 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/05cc587f82863b50b6096cc778819b35e008c76b3d582a820df75dbe199f345b/merged 2024-04-16T05:41:05.996 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/4d1fbeb092272dd4340465b4f5b973aafe131cfa5b292ad0ee1456fd29685861/merged 2024-04-16T05:41:05.996 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/c2c844a238e28e875d871851c426a44f2a1072b40182a95c74c9fb9aad937b91/merged 2024-04-16T05:41:05.997 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/597c7db60e45a0f0aa85a1fece7d7a55f037a296333d561be2ccdb2761f7e5fa/merged 2024-04-16T05:41:05.997 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/35df1eb266f0fb4246316203cb6b46d5f438c7faf3232f331702663115705b94/merged 2024-04-16T05:41:05.997 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-04-16T05:41:05.997 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T05:41:05.997 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T05:41:05.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:41:05.998 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-04-16T05:41:06.045 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-16T05:41:06.045 DEBUG:teuthology.orchestra.run.smithi122:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-16T05:41:06.050 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-04-16T05:41:06.123 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T05:41:06.165 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-04-16T05:41:06.165 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-04-16T05:41:06.165 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-04-16T05:41:06.165 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-04-16T05:41:06.165 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-04-16T05:41:06.165 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-04-16T05:41:06.165 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-04-16T05:41:06.166 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-04-16T05:41:06.166 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-04-16T05:41:06.166 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-04-16T05:41:06.166 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-04-16T05:41:06.166 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-04-16T05:41:06.166 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T05:41:06.166 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-04-16T05:41:06.166 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-04-16T05:41:06.166 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-04-16T05:41:06.166 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-04-16T05:41:06.166 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-04-16T05:41:06.166 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T05:41:06.166 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T05:41:06.166 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-04-16T05:41:06.167 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T05:41:06.167 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T05:41:06.167 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-04-16T05:41:06.167 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-04-16T05:41:06.167 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/0 2024-04-16T05:41:06.167 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T05:41:06.167 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay 2024-04-16T05:41:06.167 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/4deadc44deea52cd2034db77cfbb4d78387055e3f20ec6eedbbc750a4ba6e6e4/merged 2024-04-16T05:41:06.167 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/619822cbb0be564e77872d1a5b2246f9c86822ea3b6de049a5ae3cbdd30954b7/merged 2024-04-16T05:41:06.167 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/a85cb7c54677585c51ef3b81321e3afc02b05e9a39337ba3f5b436c4774aae8f/merged 2024-04-16T05:41:06.167 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/104c989e7579aaadd5b157ea98e406e7cb88fbe74511b7dc00d369646e95c3f8/merged 2024-04-16T05:41:06.167 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/05cc587f82863b50b6096cc778819b35e008c76b3d582a820df75dbe199f345b/merged 2024-04-16T05:41:06.167 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/4d1fbeb092272dd4340465b4f5b973aafe131cfa5b292ad0ee1456fd29685861/merged 2024-04-16T05:41:06.167 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/c2c844a238e28e875d871851c426a44f2a1072b40182a95c74c9fb9aad937b91/merged 2024-04-16T05:41:06.167 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/597c7db60e45a0f0aa85a1fece7d7a55f037a296333d561be2ccdb2761f7e5fa/merged 2024-04-16T05:41:06.168 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/35df1eb266f0fb4246316203cb6b46d5f438c7faf3232f331702663115705b94/merged 2024-04-16T05:41:06.168 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-04-16T05:41:06.168 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T05:41:06.168 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T05:41:06.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:41:06.169 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-04-16T05:41:06.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:2024-04-16T05:41:06.252+0000 7f41219d8300 -1 init, newargv = 0x556d8229e010 newargc=16 2024-04-16T05:41:06.250 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:ceph-fuse[61130]: starting ceph client 2024-04-16T05:41:06.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:FUSE library version: 2.9.9 2024-04-16T05:41:06.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:ceph-fuse[61130]: starting fuse 2024-04-16T05:41:06.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stdout:ceph_fuse: starting fuse loop 2024-04-16T05:41:06.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:2024-04-16T05:41:06.263+0000 7f41219d8300 -1 multithreaded fuse 2024-04-16T05:41:06.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 2, opcode: INIT (26), nodeid: 0, insize: 104, pid: 0 2024-04-16T05:41:06.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:INIT: 7.40 2024-04-16T05:41:06.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:flags=0x73fffffb 2024-04-16T05:41:06.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:max_readahead=0x00020000 2024-04-16T05:41:06.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: INIT: 7.19 2024-04-16T05:41:06.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: flags=0x0000043b 2024-04-16T05:41:06.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: max_readahead=0x00020000 2024-04-16T05:41:06.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: max_write=0x00020000 2024-04-16T05:41:06.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: max_background=0 2024-04-16T05:41:06.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: congestion_threshold=0 2024-04-16T05:41:06.263 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 2, success, outsize: 40 2024-04-16T05:41:06.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 4, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 61174 2024-04-16T05:41:06.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 4, success, outsize: 120 2024-04-16T05:41:07.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:06 smithi122 ceph-mon[35998]: pgmap v252: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T05:41:07.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:06 smithi027 ceph-mon[29206]: pgmap v252: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T05:41:07.354 DEBUG:teuthology.orchestra.run.smithi122:> sudo modprobe fuse 2024-04-16T05:41:07.456 DEBUG:teuthology.orchestra.run.smithi122:> cat /proc/self/mounts | awk '{print $2}' 2024-04-16T05:41:07.510 INFO:teuthology.orchestra.run.smithi122.stdout:/proc 2024-04-16T05:41:07.511 INFO:teuthology.orchestra.run.smithi122.stdout:/sys 2024-04-16T05:41:07.511 INFO:teuthology.orchestra.run.smithi122.stdout:/dev 2024-04-16T05:41:07.511 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/security 2024-04-16T05:41:07.511 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/shm 2024-04-16T05:41:07.511 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/pts 2024-04-16T05:41:07.511 INFO:teuthology.orchestra.run.smithi122.stdout:/run 2024-04-16T05:41:07.511 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/cgroup 2024-04-16T05:41:07.511 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/pstore 2024-04-16T05:41:07.511 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/bpf 2024-04-16T05:41:07.511 INFO:teuthology.orchestra.run.smithi122.stdout:/ 2024-04-16T05:41:07.511 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/selinux 2024-04-16T05:41:07.511 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T05:41:07.511 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/hugepages 2024-04-16T05:41:07.511 INFO:teuthology.orchestra.run.smithi122.stdout:/dev/mqueue 2024-04-16T05:41:07.512 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/debug 2024-04-16T05:41:07.512 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/tracing 2024-04-16T05:41:07.512 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/fs/fuse/connections 2024-04-16T05:41:07.512 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-sysctl.service 2024-04-16T05:41:07.512 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-16T05:41:07.512 INFO:teuthology.orchestra.run.smithi122.stdout:/sys/kernel/config 2024-04-16T05:41:07.512 INFO:teuthology.orchestra.run.smithi122.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-16T05:41:07.512 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/nfs/rpc_pipefs 2024-04-16T05:41:07.512 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/1000 2024-04-16T05:41:07.512 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/ceph 2024-04-16T05:41:07.512 INFO:teuthology.orchestra.run.smithi122.stdout:/run/user/0 2024-04-16T05:41:07.512 INFO:teuthology.orchestra.run.smithi122.stdout:/proc/sys/fs/binfmt_misc 2024-04-16T05:41:07.512 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay 2024-04-16T05:41:07.512 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/4deadc44deea52cd2034db77cfbb4d78387055e3f20ec6eedbbc750a4ba6e6e4/merged 2024-04-16T05:41:07.513 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/619822cbb0be564e77872d1a5b2246f9c86822ea3b6de049a5ae3cbdd30954b7/merged 2024-04-16T05:41:07.513 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/a85cb7c54677585c51ef3b81321e3afc02b05e9a39337ba3f5b436c4774aae8f/merged 2024-04-16T05:41:07.513 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/104c989e7579aaadd5b157ea98e406e7cb88fbe74511b7dc00d369646e95c3f8/merged 2024-04-16T05:41:07.513 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/05cc587f82863b50b6096cc778819b35e008c76b3d582a820df75dbe199f345b/merged 2024-04-16T05:41:07.513 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/4d1fbeb092272dd4340465b4f5b973aafe131cfa5b292ad0ee1456fd29685861/merged 2024-04-16T05:41:07.513 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/c2c844a238e28e875d871851c426a44f2a1072b40182a95c74c9fb9aad937b91/merged 2024-04-16T05:41:07.513 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/597c7db60e45a0f0aa85a1fece7d7a55f037a296333d561be2ccdb2761f7e5fa/merged 2024-04-16T05:41:07.513 INFO:teuthology.orchestra.run.smithi122.stdout:/var/lib/containers/storage/overlay/35df1eb266f0fb4246316203cb6b46d5f438c7faf3232f331702663115705b94/merged 2024-04-16T05:41:07.513 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns 2024-04-16T05:41:07.513 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T05:41:07.513 INFO:teuthology.orchestra.run.smithi122.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-16T05:41:07.513 INFO:teuthology.orchestra.run.smithi122.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-16T05:41:07.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:41:07.514 DEBUG:teuthology.orchestra.run.smithi122:> ls /sys/fs/fuse/connections 2024-04-16T05:41:07.566 INFO:teuthology.orchestra.run.smithi122.stdout:115 2024-04-16T05:41:07.566 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [115] 2024-04-16T05:41:07.566 DEBUG:teuthology.orchestra.run.smithi122:> sudo stdin-killer -- python3 -c ' 2024-04-16T05:41:07.567 DEBUG:teuthology.orchestra.run.smithi122:> import glob 2024-04-16T05:41:07.567 DEBUG:teuthology.orchestra.run.smithi122:> import re 2024-04-16T05:41:07.567 DEBUG:teuthology.orchestra.run.smithi122:> import os 2024-04-16T05:41:07.567 DEBUG:teuthology.orchestra.run.smithi122:> import subprocess 2024-04-16T05:41:07.567 DEBUG:teuthology.orchestra.run.smithi122:> 2024-04-16T05:41:07.567 DEBUG:teuthology.orchestra.run.smithi122:> def _find_admin_socket(client_name): 2024-04-16T05:41:07.567 DEBUG:teuthology.orchestra.run.smithi122:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-16T05:41:07.567 DEBUG:teuthology.orchestra.run.smithi122:> files = glob.glob(asok_path) 2024-04-16T05:41:07.567 DEBUG:teuthology.orchestra.run.smithi122:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-16T05:41:07.567 DEBUG:teuthology.orchestra.run.smithi122:> 2024-04-16T05:41:07.567 DEBUG:teuthology.orchestra.run.smithi122:> # Given a non-glob path, it better be there 2024-04-16T05:41:07.567 DEBUG:teuthology.orchestra.run.smithi122:> if "*" not in asok_path: 2024-04-16T05:41:07.567 DEBUG:teuthology.orchestra.run.smithi122:> assert(len(files) == 1) 2024-04-16T05:41:07.567 DEBUG:teuthology.orchestra.run.smithi122:> return files[0] 2024-04-16T05:41:07.567 DEBUG:teuthology.orchestra.run.smithi122:> 2024-04-16T05:41:07.567 DEBUG:teuthology.orchestra.run.smithi122:> for f in files: 2024-04-16T05:41:07.567 DEBUG:teuthology.orchestra.run.smithi122:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-16T05:41:07.568 DEBUG:teuthology.orchestra.run.smithi122:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-16T05:41:07.568 DEBUG:teuthology.orchestra.run.smithi122:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-16T05:41:07.568 DEBUG:teuthology.orchestra.run.smithi122:> contents = proc_f.read() 2024-04-16T05:41:07.568 DEBUG:teuthology.orchestra.run.smithi122:> if mountpoint in contents: 2024-04-16T05:41:07.568 DEBUG:teuthology.orchestra.run.smithi122:> return f 2024-04-16T05:41:07.568 DEBUG:teuthology.orchestra.run.smithi122:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-16T05:41:07.568 DEBUG:teuthology.orchestra.run.smithi122:> 2024-04-16T05:41:07.568 DEBUG:teuthology.orchestra.run.smithi122:> print(_find_admin_socket("client.1")) 2024-04-16T05:41:07.568 DEBUG:teuthology.orchestra.run.smithi122:> ' 2024-04-16T05:41:07.745 INFO:teuthology.orchestra.run.smithi122.stdout:/var/run/ceph/ceph-client.1.61130.asok 2024-04-16T05:41:07.750 INFO:teuthology.orchestra.run.smithi122.stderr:2024-04-16T05:41:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-16T05:41:07.764 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.61130.asok 2024-04-16T05:41:07.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:41:07.764 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.61130.asok status 2024-04-16T05:41:07.996 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2024-04-16T05:41:07.996 INFO:teuthology.orchestra.run.smithi122.stdout: "metadata": { 2024-04-16T05:41:07.996 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_sha1": "45317f95c5587af98fd8eaefff9013a4ba51d2f7", 2024-04-16T05:41:07.997 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)", 2024-04-16T05:41:07.997 INFO:teuthology.orchestra.run.smithi122.stdout: "entity_id": "1", 2024-04-16T05:41:07.997 INFO:teuthology.orchestra.run.smithi122.stdout: "hostname": "smithi122", 2024-04-16T05:41:07.997 INFO:teuthology.orchestra.run.smithi122.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-16T05:41:07.997 INFO:teuthology.orchestra.run.smithi122.stdout: "pid": "61130", 2024-04-16T05:41:07.997 INFO:teuthology.orchestra.run.smithi122.stdout: "root": "/" 2024-04-16T05:41:07.997 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-04-16T05:41:07.997 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_count": 0, 2024-04-16T05:41:07.997 INFO:teuthology.orchestra.run.smithi122.stdout: "dentry_pinned_count": 0, 2024-04-16T05:41:07.997 INFO:teuthology.orchestra.run.smithi122.stdout: "id": 24301, 2024-04-16T05:41:07.997 INFO:teuthology.orchestra.run.smithi122.stdout: "inst": { 2024-04-16T05:41:07.997 INFO:teuthology.orchestra.run.smithi122.stdout: "name": { 2024-04-16T05:41:07.997 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "client", 2024-04-16T05:41:07.997 INFO:teuthology.orchestra.run.smithi122.stdout: "num": 24301 2024-04-16T05:41:07.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-04-16T05:41:07.998 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-04-16T05:41:07.998 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-04-16T05:41:07.998 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-04-16T05:41:07.998 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 790915989 2024-04-16T05:41:07.998 INFO:teuthology.orchestra.run.smithi122.stdout: } 2024-04-16T05:41:07.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-04-16T05:41:07.998 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": { 2024-04-16T05:41:07.998 INFO:teuthology.orchestra.run.smithi122.stdout: "type": "v1", 2024-04-16T05:41:07.998 INFO:teuthology.orchestra.run.smithi122.stdout: "addr": "192.168.0.1:0", 2024-04-16T05:41:07.998 INFO:teuthology.orchestra.run.smithi122.stdout: "nonce": 790915989 2024-04-16T05:41:07.998 INFO:teuthology.orchestra.run.smithi122.stdout: }, 2024-04-16T05:41:07.998 INFO:teuthology.orchestra.run.smithi122.stdout: "inst_str": "client.24301 192.168.0.1:0/790915989", 2024-04-16T05:41:07.999 INFO:teuthology.orchestra.run.smithi122.stdout: "addr_str": "192.168.0.1:0/790915989", 2024-04-16T05:41:07.999 INFO:teuthology.orchestra.run.smithi122.stdout: "inode_count": 1, 2024-04-16T05:41:07.999 INFO:teuthology.orchestra.run.smithi122.stdout: "mds_epoch": 15, 2024-04-16T05:41:07.999 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch": 46, 2024-04-16T05:41:07.999 INFO:teuthology.orchestra.run.smithi122.stdout: "osd_epoch_barrier": 0, 2024-04-16T05:41:07.999 INFO:teuthology.orchestra.run.smithi122.stdout: "blocklisted": false, 2024-04-16T05:41:07.999 INFO:teuthology.orchestra.run.smithi122.stdout: "fs_name": "cephfs" 2024-04-16T05:41:07.999 INFO:teuthology.orchestra.run.smithi122.stdout:} 2024-04-16T05:41:08.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:41:08.011 DEBUG:teuthology.orchestra.run.smithi027:> stat --file-system '--printf=%T 2024-04-16T05:41:08.011 DEBUG:teuthology.orchestra.run.smithi027:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-16T05:41:08.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 6, opcode: STATFS (17), nodeid: 1, insize: 40, pid: 77080 2024-04-16T05:41:08.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 6, success, outsize: 96 2024-04-16T05:41:08.079 INFO:teuthology.orchestra.run.smithi027.stdout:fuseblk 2024-04-16T05:41:08.080 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-16T05:41:08.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:41:08.080 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-16T05:41:08.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 8, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77113 2024-04-16T05:41:08.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 8, success, outsize: 120 2024-04-16T05:41:08.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 10, opcode: SETATTR (4), nodeid: 1, insize: 128, pid: 77113 2024-04-16T05:41:08.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 10, success, outsize: 120 2024-04-16T05:41:08.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:41:08.196 DEBUG:teuthology.orchestra.run.smithi122:> stat --file-system '--printf=%T 2024-04-16T05:41:08.196 DEBUG:teuthology.orchestra.run.smithi122:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-16T05:41:08.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 6, opcode: STATFS (17), nodeid: 1, insize: 40, pid: 61265 2024-04-16T05:41:08.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 6, success, outsize: 96 2024-04-16T05:41:08.255 INFO:teuthology.orchestra.run.smithi122.stdout:fuseblk 2024-04-16T05:41:08.255 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-16T05:41:08.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:41:08.255 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-16T05:41:08.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 8, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 61298 2024-04-16T05:41:08.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 8, success, outsize: 120 2024-04-16T05:41:08.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 10, opcode: SETATTR (4), nodeid: 1, insize: 128, pid: 61298 2024-04-16T05:41:08.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 10, success, outsize: 120 2024-04-16T05:41:08.355 INFO:teuthology.run_tasks:Running task print... 2024-04-16T05:41:08.365 INFO:teuthology.task.print:**** done client 2024-04-16T05:41:08.366 INFO:teuthology.run_tasks:Running task parallel... 2024-04-16T05:41:08.378 INFO:teuthology.task.parallel:starting parallel... 2024-04-16T05:41:08.378 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-16T05:41:08.379 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-16T05:41:08.379 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-16T05:41:08.379 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph config set mgr mgr/orchestrator/fail_fs false || true' 2024-04-16T05:41:08.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:08 smithi027 ceph-mon[29206]: pgmap v253: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T05:41:08.910 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:41:09.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:08 smithi122 ceph-mon[35998]: pgmap v253: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T05:41:10.969 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:10 smithi027 ceph-mon[29206]: pgmap v254: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-16T05:41:11.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:10 smithi122 ceph-mon[35998]: pgmap v254: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-04-16T05:41:11.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.255+0000 7f341b200700 1 -- 172.21.15.27:0/1160151128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f341c103270 msgr2=0x7f341c103680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:11.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.255+0000 7f341b200700 1 --2- 172.21.15.27:0/1160151128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f341c103270 0x7f341c103680 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f340c004940 tx=0x7f340c009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:11.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.255+0000 7f341b200700 1 -- 172.21.15.27:0/1160151128 shutdown_connections 2024-04-16T05:41:11.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.255+0000 7f341b200700 1 --2- 172.21.15.27:0/1160151128 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f341c104520 0x7f341c104990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:11.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.255+0000 7f341b200700 1 --2- 172.21.15.27:0/1160151128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f341c103270 0x7f341c103680 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:11.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.255+0000 7f341b200700 1 -- 172.21.15.27:0/1160151128 >> 172.21.15.27:0/1160151128 conn(0x7f341c072530 msgr2=0x7f341c074940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:11.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.256+0000 7f341b200700 1 -- 172.21.15.27:0/1160151128 shutdown_connections 2024-04-16T05:41:11.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.256+0000 7f341b200700 1 -- 172.21.15.27:0/1160151128 wait complete. 2024-04-16T05:41:11.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.257+0000 7f341b200700 1 Processor -- start 2024-04-16T05:41:11.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.257+0000 7f341b200700 1 -- start start 2024-04-16T05:41:11.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.257+0000 7f341b200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f341c103270 0x7f341c113f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:11.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.257+0000 7f341b200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f341c104520 0x7f341c114440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:11.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.257+0000 7f341b200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c114a40 con 0x7f341c103270 2024-04-16T05:41:11.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.257+0000 7f341b200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c115180 con 0x7f341c104520 2024-04-16T05:41:11.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.258+0000 7f3419e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f341c103270 0x7f341c113f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:11.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.258+0000 7f3419e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f341c103270 0x7f341c113f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33382/0 (socket says 172.21.15.27:33382) 2024-04-16T05:41:11.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.258+0000 7f3419e00700 1 -- 172.21.15.27:0/1193383916 learned_addr learned my addr 172.21.15.27:0/1193383916 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:41:11.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.258+0000 7f3413e00700 1 --2- 172.21.15.27:0/1193383916 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f341c104520 0x7f341c114440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:11.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.258+0000 7f3419e00700 1 -- 172.21.15.27:0/1193383916 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f341c104520 msgr2=0x7f341c114440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:11.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.258+0000 7f3419e00700 1 --2- 172.21.15.27:0/1193383916 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f341c104520 0x7f341c114440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:11.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.258+0000 7f3419e00700 1 -- 172.21.15.27:0/1193383916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f340c005040 con 0x7f341c103270 2024-04-16T05:41:11.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.259+0000 7f3419e00700 1 --2- 172.21.15.27:0/1193383916 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f341c103270 0x7f341c113f00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f340c00b3b0 tx=0x7f340c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:11.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.259+0000 7f3412000700 1 -- 172.21.15.27:0/1193383916 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f340c00e040 con 0x7f341c103270 2024-04-16T05:41:11.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.259+0000 7f3412000700 1 -- 172.21.15.27:0/1193383916 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f340c00bbf0 con 0x7f341c103270 2024-04-16T05:41:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.259+0000 7f341b200700 1 -- 172.21.15.27:0/1193383916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f341c115400 con 0x7f341c103270 2024-04-16T05:41:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.259+0000 7f3412000700 1 -- 172.21.15.27:0/1193383916 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f340c01c630 con 0x7f341c103270 2024-04-16T05:41:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.260+0000 7f341b200700 1 -- 172.21.15.27:0/1193383916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f341c1159c0 con 0x7f341c103270 2024-04-16T05:41:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.261+0000 7f341b200700 1 -- 172.21.15.27:0/1193383916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f341c0760a0 con 0x7f341c103270 2024-04-16T05:41:11.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.262+0000 7f3412000700 1 -- 172.21.15.27:0/1193383916 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f340c00bd60 con 0x7f341c103270 2024-04-16T05:41:11.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.264+0000 7f3412000700 1 --2- 172.21.15.27:0/1193383916 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f34000692a0 0x7f340006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:11.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.264+0000 7f3412000700 1 -- 172.21.15.27:0/1193383916 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f340c087fb0 con 0x7f341c103270 2024-04-16T05:41:11.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.264+0000 7f3413e00700 1 --2- 172.21.15.27:0/1193383916 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f34000692a0 0x7f340006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:11.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.265+0000 7f3413e00700 1 --2- 172.21.15.27:0/1193383916 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f34000692a0 0x7f340006b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3404004f40 tx=0x7f340400b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:11.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.268+0000 7f3412000700 1 -- 172.21.15.27:0/1193383916 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f340c0ba250 con 0x7f341c103270 2024-04-16T05:41:11.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.486+0000 7f341b200700 1 -- 172.21.15.27:0/1193383916 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mgr/orchestrator/fail_fs}] v 0) v1 -- 0x7f341c04bed0 con 0x7f341c103270 2024-04-16T05:41:11.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:11.487+0000 7f3412000700 1 -- 172.21.15.27:0/1193383916 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/orchestrator/fail_fs}]=-22 unrecognized config option 'mgr/orchestrator/fail_fs' v21)=-22 unrecognized config option 'mgr/orchestrator/fail_fs' v21) v1 ==== 179+0+0 (secure 0 0 0) 0x7f340c057c80 con 0x7f341c103270 2024-04-16T05:41:11.489 INFO:teuthology.orchestra.run.smithi027.stderr:Error EINVAL: unrecognized config option 'mgr/orchestrator/fail_fs' 2024-04-16T05:41:12.346 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-16T05:41:12.347 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-16T05:41:12.347 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-16T05:41:12.833 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:41:12.979 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:12 smithi027 ceph-mon[29206]: pgmap v255: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T05:41:13.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:12 smithi122 ceph-mon[35998]: pgmap v255: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-04-16T05:41:15.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:14 smithi122 ceph-mon[35998]: pgmap v256: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T05:41:15.045 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:14 smithi027 ceph-mon[29206]: pgmap v256: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T05:41:15.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.228+0000 7fadf1600700 1 -- 172.21.15.27:0/2055541965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadec0fed80 msgr2=0x7fadec0faec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:15.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.228+0000 7fadf1600700 1 --2- 172.21.15.27:0/2055541965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadec0fed80 0x7fadec0faec0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fadd4004940 tx=0x7fadd4009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:15.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.228+0000 7fadf1600700 1 -- 172.21.15.27:0/2055541965 shutdown_connections 2024-04-16T05:41:15.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.228+0000 7fadf1600700 1 --2- 172.21.15.27:0/2055541965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadec0fed80 0x7fadec0faec0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:15.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.228+0000 7fadf1600700 1 --2- 172.21.15.27:0/2055541965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadec0fe430 0x7fadec0fe840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:15.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.228+0000 7fadf1600700 1 -- 172.21.15.27:0/2055541965 >> 172.21.15.27:0/2055541965 conn(0x7fadec0f6bd0 msgr2=0x7fadec0f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:15.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.228+0000 7fadf1600700 1 -- 172.21.15.27:0/2055541965 shutdown_connections 2024-04-16T05:41:15.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.228+0000 7fadf1600700 1 -- 172.21.15.27:0/2055541965 wait complete. 2024-04-16T05:41:15.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.230+0000 7fadf1600700 1 Processor -- start 2024-04-16T05:41:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.230+0000 7fadf1600700 1 -- start start 2024-04-16T05:41:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.230+0000 7fadf1600700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadec0fe430 0x7fadec10fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.230+0000 7fadf1600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadec0fed80 0x7fadec1100f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.230+0000 7fadf1600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadec1106f0 con 0x7fadec0fed80 2024-04-16T05:41:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.230+0000 7fadf1600700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadec110860 con 0x7fadec0fe430 2024-04-16T05:41:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.231+0000 7fadea000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadec0fed80 0x7fadec1100f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.231+0000 7fadea000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadec0fed80 0x7fadec1100f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59726/0 (socket says 172.21.15.27:59726) 2024-04-16T05:41:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.231+0000 7fadea000700 1 -- 172.21.15.27:0/3039026264 learned_addr learned my addr 172.21.15.27:0/3039026264 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:41:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.231+0000 7fadeaa00700 1 --2- 172.21.15.27:0/3039026264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadec0fe430 0x7fadec10fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:15.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.231+0000 7fadea000700 1 -- 172.21.15.27:0/3039026264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadec0fe430 msgr2=0x7fadec10fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:15.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.231+0000 7fadea000700 1 --2- 172.21.15.27:0/3039026264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadec0fe430 0x7fadec10fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:15.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.231+0000 7fadea000700 1 -- 172.21.15.27:0/3039026264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd4005040 con 0x7fadec0fed80 2024-04-16T05:41:15.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.231+0000 7fadea000700 1 --2- 172.21.15.27:0/3039026264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadec0fed80 0x7fadec1100f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fadd40090d0 tx=0x7fadd4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:15.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.232+0000 7fade2a00700 1 -- 172.21.15.27:0/3039026264 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadd400e040 con 0x7fadec0fed80 2024-04-16T05:41:15.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.232+0000 7fade2a00700 1 -- 172.21.15.27:0/3039026264 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fadd4004290 con 0x7fadec0fed80 2024-04-16T05:41:15.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.232+0000 7fade2a00700 1 -- 172.21.15.27:0/3039026264 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadd401c3f0 con 0x7fadec0fed80 2024-04-16T05:41:15.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.232+0000 7fadf1600700 1 -- 172.21.15.27:0/3039026264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadec1151f0 con 0x7fadec0fed80 2024-04-16T05:41:15.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.232+0000 7fadf1600700 1 -- 172.21.15.27:0/3039026264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadec1157b0 con 0x7fadec0fed80 2024-04-16T05:41:15.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.233+0000 7fadf1600700 1 -- 172.21.15.27:0/3039026264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadec101b70 con 0x7fadec0fed80 2024-04-16T05:41:15.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.234+0000 7fade2a00700 1 -- 172.21.15.27:0/3039026264 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fadd400bbf0 con 0x7fadec0fed80 2024-04-16T05:41:15.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.236+0000 7fade2a00700 1 --2- 172.21.15.27:0/3039026264 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fadd00692a0 0x7fadd006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:15.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.236+0000 7fade2a00700 1 -- 172.21.15.27:0/3039026264 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fadd4088510 con 0x7fadec0fed80 2024-04-16T05:41:15.235 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.236+0000 7fadeaa00700 1 --2- 172.21.15.27:0/3039026264 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fadd00692a0 0x7fadd006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:15.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.237+0000 7fadeaa00700 1 --2- 172.21.15.27:0/3039026264 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fadd00692a0 0x7fadd006b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7faddc007fd0 tx=0x7faddc009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:15.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.241+0000 7fade2a00700 1 -- 172.21.15.27:0/3039026264 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fadd4058260 con 0x7fadec0fed80 2024-04-16T05:41:15.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.459+0000 7fadf1600700 1 -- 172.21.15.27:0/3039026264 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fadec04c710 con 0x7fadec0fed80 2024-04-16T05:41:15.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.460+0000 7fade2a00700 1 -- 172.21.15.27:0/3039026264 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7fadec04c710 con 0x7fadec0fed80 2024-04-16T05:41:15.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.463+0000 7fadf1600700 1 -- 172.21.15.27:0/3039026264 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fadd00692a0 msgr2=0x7fadd006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:15.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.463+0000 7fadf1600700 1 --2- 172.21.15.27:0/3039026264 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fadd00692a0 0x7fadd006b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7faddc007fd0 tx=0x7faddc009040 comp rx=0 tx=0).stop 2024-04-16T05:41:15.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.463+0000 7fadf1600700 1 -- 172.21.15.27:0/3039026264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadec0fed80 msgr2=0x7fadec1100f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:15.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.463+0000 7fadf1600700 1 --2- 172.21.15.27:0/3039026264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadec0fed80 0x7fadec1100f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fadd40090d0 tx=0x7fadd4009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:15.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.463+0000 7fadf1600700 1 -- 172.21.15.27:0/3039026264 shutdown_connections 2024-04-16T05:41:15.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.463+0000 7fadf1600700 1 --2- 172.21.15.27:0/3039026264 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fadd00692a0 0x7fadd006b750 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:15.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.463+0000 7fadf1600700 1 --2- 172.21.15.27:0/3039026264 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadec0fe430 0x7fadec10fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:15.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.463+0000 7fadf1600700 1 --2- 172.21.15.27:0/3039026264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadec0fed80 0x7fadec1100f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:15.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.463+0000 7fadf1600700 1 -- 172.21.15.27:0/3039026264 >> 172.21.15.27:0/3039026264 conn(0x7fadec0f6bd0 msgr2=0x7fadec0f8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:15.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.463+0000 7fadf1600700 1 -- 172.21.15.27:0/3039026264 shutdown_connections 2024-04-16T05:41:15.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:15.463+0000 7fadf1600700 1 -- 172.21.15.27:0/3039026264 wait complete. 2024-04-16T05:41:16.363 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-16T05:41:16.883 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:41:17.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:16 smithi122 ceph-mon[35998]: pgmap v257: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T05:41:17.039 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:16 smithi027 ceph-mon[29206]: pgmap v257: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T05:41:18.929 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:18 smithi027 ceph-mon[29206]: pgmap v258: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T05:41:19.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:18 smithi122 ceph-mon[35998]: pgmap v258: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T05:41:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.226+0000 7f418f200700 1 -- 172.21.15.27:0/763960823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41880fdff0 msgr2=0x7f41880fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.226+0000 7f418f200700 1 --2- 172.21.15.27:0/763960823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41880fdff0 0x7f41880fe460 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4174004940 tx=0x7f4174009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.226+0000 7f418f200700 1 -- 172.21.15.27:0/763960823 shutdown_connections 2024-04-16T05:41:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.226+0000 7f418f200700 1 --2- 172.21.15.27:0/763960823 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41880fdff0 0x7f41880fe460 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.226+0000 7f418f200700 1 --2- 172.21.15.27:0/763960823 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41880fce80 0x7f41880fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.226+0000 7f418f200700 1 -- 172.21.15.27:0/763960823 >> 172.21.15.27:0/763960823 conn(0x7f41880f8580 msgr2=0x7f41880fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:19.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.226+0000 7f418f200700 1 -- 172.21.15.27:0/763960823 shutdown_connections 2024-04-16T05:41:19.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.226+0000 7f418f200700 1 -- 172.21.15.27:0/763960823 wait complete. 2024-04-16T05:41:19.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.227+0000 7f418f200700 1 Processor -- start 2024-04-16T05:41:19.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.227+0000 7f418f200700 1 -- start start 2024-04-16T05:41:19.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.228+0000 7f418f200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41880fce80 0x7f418810d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:19.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.228+0000 7f418f200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41880fdff0 0x7f418810de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:19.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.228+0000 7f418f200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418810e450 con 0x7f41880fdff0 2024-04-16T05:41:19.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.228+0000 7f418f200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418810e5c0 con 0x7f41880fce80 2024-04-16T05:41:19.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.228+0000 7f4187400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41880fdff0 0x7f418810de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:19.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.228+0000 7f4187400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41880fdff0 0x7f418810de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59736/0 (socket says 172.21.15.27:59736) 2024-04-16T05:41:19.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.228+0000 7f4187400700 1 -- 172.21.15.27:0/539543519 learned_addr learned my addr 172.21.15.27:0/539543519 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:41:19.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.229+0000 7f4187e00700 1 --2- 172.21.15.27:0/539543519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41880fce80 0x7f418810d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:19.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.229+0000 7f4187400700 1 -- 172.21.15.27:0/539543519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41880fce80 msgr2=0x7f418810d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:19.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.229+0000 7f4187400700 1 --2- 172.21.15.27:0/539543519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41880fce80 0x7f418810d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:19.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.229+0000 7f4187400700 1 -- 172.21.15.27:0/539543519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4174005040 con 0x7f41880fdff0 2024-04-16T05:41:19.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.229+0000 7f4187400700 1 --2- 172.21.15.27:0/539543519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41880fdff0 0x7f418810de50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f417400b3b0 tx=0x7f4174009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:19.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.229+0000 7f417fe00700 1 -- 172.21.15.27:0/539543519 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f417400e040 con 0x7f41880fdff0 2024-04-16T05:41:19.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.230+0000 7f417fe00700 1 -- 172.21.15.27:0/539543519 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4174004290 con 0x7f41880fdff0 2024-04-16T05:41:19.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.230+0000 7f417fe00700 1 -- 172.21.15.27:0/539543519 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f417401c630 con 0x7f41880fdff0 2024-04-16T05:41:19.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.230+0000 7f418f200700 1 -- 172.21.15.27:0/539543519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4188112fc0 con 0x7f41880fdff0 2024-04-16T05:41:19.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.230+0000 7f418f200700 1 -- 172.21.15.27:0/539543519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4188113550 con 0x7f41880fdff0 2024-04-16T05:41:19.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.231+0000 7f418f200700 1 -- 172.21.15.27:0/539543519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f418804c710 con 0x7f41880fdff0 2024-04-16T05:41:19.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.232+0000 7f417fe00700 1 -- 172.21.15.27:0/539543519 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f417400bbf0 con 0x7f41880fdff0 2024-04-16T05:41:19.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.233+0000 7f417fe00700 1 --2- 172.21.15.27:0/539543519 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f416c0692a0 0x7f416c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:19.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.233+0000 7f417fe00700 1 -- 172.21.15.27:0/539543519 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f41740886d0 con 0x7f41880fdff0 2024-04-16T05:41:19.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.234+0000 7f4187e00700 1 --2- 172.21.15.27:0/539543519 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f416c0692a0 0x7f416c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:19.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.234+0000 7f4187e00700 1 --2- 172.21.15.27:0/539543519 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f416c0692a0 0x7f416c06b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4178007fd0 tx=0x7f4178009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:19.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.239+0000 7f417fe00700 1 -- 172.21.15.27:0/539543519 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f41740583a0 con 0x7f41880fdff0 2024-04-16T05:41:19.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.456+0000 7f418f200700 1 -- 172.21.15.27:0/539543519 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f4188113990 con 0x7f41880fdff0 2024-04-16T05:41:19.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.456+0000 7f417fe00700 1 -- 172.21.15.27:0/539543519 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f4174057f30 con 0x7f41880fdff0 2024-04-16T05:41:19.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.460+0000 7f418f200700 1 -- 172.21.15.27:0/539543519 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f416c0692a0 msgr2=0x7f416c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:19.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.460+0000 7f418f200700 1 --2- 172.21.15.27:0/539543519 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f416c0692a0 0x7f416c06b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4178007fd0 tx=0x7f4178009040 comp rx=0 tx=0).stop 2024-04-16T05:41:19.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.460+0000 7f418f200700 1 -- 172.21.15.27:0/539543519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41880fdff0 msgr2=0x7f418810de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:19.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.460+0000 7f418f200700 1 --2- 172.21.15.27:0/539543519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41880fdff0 0x7f418810de50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f417400b3b0 tx=0x7f4174009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:19.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.460+0000 7f418f200700 1 -- 172.21.15.27:0/539543519 shutdown_connections 2024-04-16T05:41:19.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.460+0000 7f418f200700 1 --2- 172.21.15.27:0/539543519 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f416c0692a0 0x7f416c06b750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:19.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.460+0000 7f418f200700 1 --2- 172.21.15.27:0/539543519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41880fce80 0x7f418810d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:19.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.460+0000 7f418f200700 1 --2- 172.21.15.27:0/539543519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41880fdff0 0x7f418810de50 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:19.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.460+0000 7f418f200700 1 -- 172.21.15.27:0/539543519 >> 172.21.15.27:0/539543519 conn(0x7f41880f8580 msgr2=0x7f41880f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:19.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.460+0000 7f418f200700 1 -- 172.21.15.27:0/539543519 shutdown_connections 2024-04-16T05:41:19.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:19.460+0000 7f418f200700 1 -- 172.21.15.27:0/539543519 wait complete. 2024-04-16T05:41:20.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:19 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:41:20.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:19 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:41:20.380 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-16T05:41:20.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:20 smithi122 ceph-mon[35998]: pgmap v259: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T05:41:20.894 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:41:21.042 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:20 smithi027 ceph-mon[29206]: pgmap v259: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T05:41:22.754 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:22 smithi027 ceph-mon[29206]: pgmap v260: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-16T05:41:23.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:22 smithi122 ceph-mon[35998]: pgmap v260: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-16T05:41:23.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.055+0000 7fe5ef200700 1 -- 172.21.15.27:0/3808782681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5f00fded0 msgr2=0x7fe5f00fe340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:23.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.055+0000 7fe5ef200700 1 --2- 172.21.15.27:0/3808782681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5f00fded0 0x7fe5f00fe340 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc004940 tx=0x7fe5dc009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:23.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.055+0000 7fe5ef200700 1 -- 172.21.15.27:0/3808782681 shutdown_connections 2024-04-16T05:41:23.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.055+0000 7fe5ef200700 1 --2- 172.21.15.27:0/3808782681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5f00fded0 0x7fe5f00fe340 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:23.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.055+0000 7fe5ef200700 1 --2- 172.21.15.27:0/3808782681 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5f00fcc20 0x7fe5f00fd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:23.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.055+0000 7fe5ef200700 1 -- 172.21.15.27:0/3808782681 >> 172.21.15.27:0/3808782681 conn(0x7fe5f00f83a0 msgr2=0x7fe5f00fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:23.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.055+0000 7fe5ef200700 1 -- 172.21.15.27:0/3808782681 shutdown_connections 2024-04-16T05:41:23.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.055+0000 7fe5ef200700 1 -- 172.21.15.27:0/3808782681 wait complete. 2024-04-16T05:41:23.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.056+0000 7fe5ef200700 1 Processor -- start 2024-04-16T05:41:23.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.057+0000 7fe5ef200700 1 -- start start 2024-04-16T05:41:23.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.057+0000 7fe5ef200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5f00fcc20 0x7fe5f010fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:23.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.057+0000 7fe5ef200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5f00fded0 0x7fe5f010ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:23.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.057+0000 7fe5ef200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f0110590 con 0x7fe5f00fded0 2024-04-16T05:41:23.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.057+0000 7fe5ef200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f0110de0 con 0x7fe5f00fcc20 2024-04-16T05:41:23.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.058+0000 7fe5ed400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5f00fded0 0x7fe5f010ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:23.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.058+0000 7fe5ed400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5f00fded0 0x7fe5f010ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59752/0 (socket says 172.21.15.27:59752) 2024-04-16T05:41:23.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.058+0000 7fe5ed400700 1 -- 172.21.15.27:0/1199649231 learned_addr learned my addr 172.21.15.27:0/1199649231 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:41:23.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.058+0000 7fe5ede00700 1 --2- 172.21.15.27:0/1199649231 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5f00fcc20 0x7fe5f010fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:23.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.058+0000 7fe5ed400700 1 -- 172.21.15.27:0/1199649231 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5f00fcc20 msgr2=0x7fe5f010fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:23.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.058+0000 7fe5ed400700 1 --2- 172.21.15.27:0/1199649231 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5f00fcc20 0x7fe5f010fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:23.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.058+0000 7fe5ed400700 1 -- 172.21.15.27:0/1199649231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5dc005040 con 0x7fe5f00fded0 2024-04-16T05:41:23.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.059+0000 7fe5ed400700 1 --2- 172.21.15.27:0/1199649231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5f00fded0 0x7fe5f010ff90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc009150 tx=0x7fe5dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.059+0000 7fe5e6000700 1 -- 172.21.15.27:0/1199649231 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5dc00e040 con 0x7fe5f00fded0 2024-04-16T05:41:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.059+0000 7fe5e6000700 1 -- 172.21.15.27:0/1199649231 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe5dc00bbf0 con 0x7fe5f00fded0 2024-04-16T05:41:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.059+0000 7fe5ef200700 1 -- 172.21.15.27:0/1199649231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f0111060 con 0x7fe5f00fded0 2024-04-16T05:41:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.059+0000 7fe5e6000700 1 -- 172.21.15.27:0/1199649231 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5dc01c630 con 0x7fe5f00fded0 2024-04-16T05:41:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.059+0000 7fe5ef200700 1 -- 172.21.15.27:0/1199649231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f01115a0 con 0x7fe5f00fded0 2024-04-16T05:41:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.061+0000 7fe5ef200700 1 -- 172.21.15.27:0/1199649231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f0101860 con 0x7fe5f00fded0 2024-04-16T05:41:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.062+0000 7fe5e6000700 1 -- 172.21.15.27:0/1199649231 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fe5dc00bd60 con 0x7fe5f00fded0 2024-04-16T05:41:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.063+0000 7fe5e6000700 1 --2- 172.21.15.27:0/1199649231 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fe5d40692a0 0x7fe5d406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.063+0000 7fe5e6000700 1 -- 172.21.15.27:0/1199649231 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe5dc0885e0 con 0x7fe5f00fded0 2024-04-16T05:41:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.063+0000 7fe5ede00700 1 --2- 172.21.15.27:0/1199649231 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fe5d40692a0 0x7fe5d406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.064+0000 7fe5ede00700 1 --2- 172.21.15.27:0/1199649231 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fe5d40692a0 0x7fe5d406b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0004990 tx=0x7fe5e0009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:23.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.068+0000 7fe5e6000700 1 -- 172.21.15.27:0/1199649231 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fe5dc058330 con 0x7fe5f00fded0 2024-04-16T05:41:23.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.286+0000 7fe5ef200700 1 -- 172.21.15.27:0/1199649231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7fe5f00fc020 con 0x7fe5f00fded0 2024-04-16T05:41:23.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.287+0000 7fe5e6000700 1 -- 172.21.15.27:0/1199649231 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7fe5f00fc020 con 0x7fe5f00fded0 2024-04-16T05:41:23.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.291+0000 7fe5ef200700 1 -- 172.21.15.27:0/1199649231 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fe5d40692a0 msgr2=0x7fe5d406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:23.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.291+0000 7fe5ef200700 1 --2- 172.21.15.27:0/1199649231 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fe5d40692a0 0x7fe5d406b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0004990 tx=0x7fe5e0009ee0 comp rx=0 tx=0).stop 2024-04-16T05:41:23.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.291+0000 7fe5ef200700 1 -- 172.21.15.27:0/1199649231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5f00fded0 msgr2=0x7fe5f010ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:23.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.291+0000 7fe5ef200700 1 --2- 172.21.15.27:0/1199649231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5f00fded0 0x7fe5f010ff90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc009150 tx=0x7fe5dc009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:23.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.291+0000 7fe5ef200700 1 -- 172.21.15.27:0/1199649231 shutdown_connections 2024-04-16T05:41:23.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.291+0000 7fe5ef200700 1 --2- 172.21.15.27:0/1199649231 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fe5d40692a0 0x7fe5d406b750 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:23.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.291+0000 7fe5ef200700 1 --2- 172.21.15.27:0/1199649231 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5f00fcc20 0x7fe5f010fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:23.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.291+0000 7fe5ef200700 1 --2- 172.21.15.27:0/1199649231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe5f00fded0 0x7fe5f010ff90 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:23.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.291+0000 7fe5ef200700 1 -- 172.21.15.27:0/1199649231 >> 172.21.15.27:0/1199649231 conn(0x7fe5f00f83a0 msgr2=0x7fe5f00f99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:23.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.291+0000 7fe5ef200700 1 -- 172.21.15.27:0/1199649231 shutdown_connections 2024-04-16T05:41:23.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:23.291+0000 7fe5ef200700 1 -- 172.21.15.27:0/1199649231 wait complete. 2024-04-16T05:41:24.257 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-16T05:41:24.796 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:41:25.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:24 smithi027 ceph-mon[29206]: pgmap v261: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-16T05:41:25.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:24 smithi122 ceph-mon[35998]: pgmap v261: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-04-16T05:41:26.023 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:25 smithi027 ceph-mon[29206]: pgmap v262: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:26.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:25 smithi122 ceph-mon[35998]: pgmap v262: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:26.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.949+0000 7effc7e00700 1 -- 172.21.15.27:0/168906978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc8100090 msgr2=0x7effc8100500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.949+0000 7effc7e00700 1 --2- 172.21.15.27:0/168906978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc8100090 0x7effc8100500 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7effb4004940 tx=0x7effb4009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.949+0000 7effc7e00700 1 -- 172.21.15.27:0/168906978 shutdown_connections 2024-04-16T05:41:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.949+0000 7effc7e00700 1 --2- 172.21.15.27:0/168906978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc8100090 0x7effc8100500 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.949+0000 7effc7e00700 1 --2- 172.21.15.27:0/168906978 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7effc80fee90 0x7effc80ff2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.949+0000 7effc7e00700 1 -- 172.21.15.27:0/168906978 >> 172.21.15.27:0/168906978 conn(0x7effc80fa5d0 msgr2=0x7effc80fc9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.949+0000 7effc7e00700 1 -- 172.21.15.27:0/168906978 shutdown_connections 2024-04-16T05:41:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.950+0000 7effc7e00700 1 -- 172.21.15.27:0/168906978 wait complete. 2024-04-16T05:41:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.950+0000 7effc7e00700 1 Processor -- start 2024-04-16T05:41:26.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.951+0000 7effc7e00700 1 -- start start 2024-04-16T05:41:26.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.951+0000 7effc7e00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7effc80fee90 0x7effc810fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:26.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.951+0000 7effc7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc8100090 0x7effc8110000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:26.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.951+0000 7effc7e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc8110600 con 0x7effc8100090 2024-04-16T05:41:26.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.951+0000 7effc7e00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc8110e00 con 0x7effc80fee90 2024-04-16T05:41:26.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.951+0000 7effc6000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc8100090 0x7effc8110000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:26.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.951+0000 7effc6000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc8100090 0x7effc8110000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53882/0 (socket says 172.21.15.27:53882) 2024-04-16T05:41:26.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.951+0000 7effc6000700 1 -- 172.21.15.27:0/2111327693 learned_addr learned my addr 172.21.15.27:0/2111327693 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:41:26.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.952+0000 7effc6a00700 1 --2- 172.21.15.27:0/2111327693 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7effc80fee90 0x7effc810fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:26.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.952+0000 7effc6000700 1 -- 172.21.15.27:0/2111327693 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7effc80fee90 msgr2=0x7effc810fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:26.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.952+0000 7effc6000700 1 --2- 172.21.15.27:0/2111327693 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7effc80fee90 0x7effc810fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:26.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.952+0000 7effc6000700 1 -- 172.21.15.27:0/2111327693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb4005040 con 0x7effc8100090 2024-04-16T05:41:26.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.952+0000 7effc6000700 1 --2- 172.21.15.27:0/2111327693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc8100090 0x7effc8110000 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7effb400b3b0 tx=0x7effb4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:26.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.952+0000 7effbea00700 1 -- 172.21.15.27:0/2111327693 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effb400e040 con 0x7effc8100090 2024-04-16T05:41:26.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.953+0000 7effbea00700 1 -- 172.21.15.27:0/2111327693 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effb4004290 con 0x7effc8100090 2024-04-16T05:41:26.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.953+0000 7effbea00700 1 -- 172.21.15.27:0/2111327693 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7effb401c630 con 0x7effc8100090 2024-04-16T05:41:26.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.953+0000 7effc7e00700 1 -- 172.21.15.27:0/2111327693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc8111080 con 0x7effc8100090 2024-04-16T05:41:26.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.953+0000 7effc7e00700 1 -- 172.21.15.27:0/2111327693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc81115c0 con 0x7effc8100090 2024-04-16T05:41:26.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.954+0000 7effc7e00700 1 -- 172.21.15.27:0/2111327693 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc805f5f0 con 0x7effc8100090 2024-04-16T05:41:26.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.955+0000 7effbea00700 1 -- 172.21.15.27:0/2111327693 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7effb400bbf0 con 0x7effc8100090 2024-04-16T05:41:26.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.956+0000 7effbea00700 1 --2- 172.21.15.27:0/2111327693 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7effac0692b0 0x7effac06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:26.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.957+0000 7effbea00700 1 -- 172.21.15.27:0/2111327693 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7effb4088660 con 0x7effc8100090 2024-04-16T05:41:26.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.957+0000 7effc6a00700 1 --2- 172.21.15.27:0/2111327693 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7effac0692b0 0x7effac06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:26.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.957+0000 7effc6a00700 1 --2- 172.21.15.27:0/2111327693 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7effac0692b0 0x7effac06b760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7effb8007fd0 tx=0x7effb8009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:26.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:26.962+0000 7effbea00700 1 -- 172.21.15.27:0/2111327693 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7effb40583b0 con 0x7effc8100090 2024-04-16T05:41:27.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:27.194+0000 7effc7e00700 1 -- 172.21.15.27:0/2111327693 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838", "target": ["mon-mgr", ""]}) v1 -- 0x7effc8111aa0 con 0x7effac0692b0 2024-04-16T05:41:27.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:27.207+0000 7effbea00700 1 -- 172.21.15.27:0/2111327693 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7effc8111aa0 con 0x7effac0692b0 2024-04-16T05:41:27.206 INFO:teuthology.orchestra.run.smithi027.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T05:41:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:27.210+0000 7effc7e00700 1 -- 172.21.15.27:0/2111327693 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7effac0692b0 msgr2=0x7effac06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:27.210+0000 7effc7e00700 1 --2- 172.21.15.27:0/2111327693 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7effac0692b0 0x7effac06b760 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7effb8007fd0 tx=0x7effb8009040 comp rx=0 tx=0).stop 2024-04-16T05:41:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:27.210+0000 7effc7e00700 1 -- 172.21.15.27:0/2111327693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc8100090 msgr2=0x7effc8110000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:27.210+0000 7effc7e00700 1 --2- 172.21.15.27:0/2111327693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc8100090 0x7effc8110000 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7effb400b3b0 tx=0x7effb4009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:27.210+0000 7effc7e00700 1 -- 172.21.15.27:0/2111327693 shutdown_connections 2024-04-16T05:41:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:27.210+0000 7effc7e00700 1 --2- 172.21.15.27:0/2111327693 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7effac0692b0 0x7effac06b760 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:27.210+0000 7effc7e00700 1 --2- 172.21.15.27:0/2111327693 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7effc80fee90 0x7effc810fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:27.210+0000 7effc7e00700 1 --2- 172.21.15.27:0/2111327693 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc8100090 0x7effc8110000 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:27.210+0000 7effc7e00700 1 -- 172.21.15.27:0/2111327693 >> 172.21.15.27:0/2111327693 conn(0x7effc80fa5d0 msgr2=0x7effc80fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:27.211+0000 7effc7e00700 1 -- 172.21.15.27:0/2111327693 shutdown_connections 2024-04-16T05:41:27.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:27.211+0000 7effc7e00700 1 -- 172.21.15.27:0/2111327693 wait complete. 2024-04-16T05:41:28.061 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-16T05:41:28.062 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-16T05:41:28.062 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-16T05:41:28.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:28 smithi027 ceph-mon[29206]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:41:28.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:28 smithi027 ceph-mon[29206]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T05:41:28.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:28 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:41:28.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:28 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:41:28.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:28 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:41:28.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:28 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:41:28.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:28 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:41:28.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:28 smithi027 ceph-mon[29206]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T05:41:28.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:28 smithi027 ceph-mon[29206]: pgmap v263: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:28.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:28 smithi122 ceph-mon[35998]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:41:28.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:28 smithi122 ceph-mon[35998]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T05:41:28.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:28 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:41:28.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:28 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:41:28.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:28 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:41:28.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:28 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:41:28.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:28 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:41:28.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:28 smithi122 ceph-mon[35998]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 2024-04-16T05:41:28.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:28 smithi122 ceph-mon[35998]: pgmap v263: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:28.549 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:41:30.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:30 smithi027 ceph-mon[29206]: pgmap v264: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:31.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:30 smithi122 ceph-mon[35998]: pgmap v264: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:31.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.075+0000 7fb78ea00700 1 -- 172.21.15.27:0/803646749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7880fcc30 msgr2=0x7fb7880fd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:31.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.075+0000 7fb78ea00700 1 --2- 172.21.15.27:0/803646749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7880fcc30 0x7fb7880fd040 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb77c004940 tx=0x7fb77c009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:31.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.075+0000 7fb78ea00700 1 -- 172.21.15.27:0/803646749 shutdown_connections 2024-04-16T05:41:31.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.075+0000 7fb78ea00700 1 --2- 172.21.15.27:0/803646749 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7880fdee0 0x7fb7880fe350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:31.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.075+0000 7fb78ea00700 1 --2- 172.21.15.27:0/803646749 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7880fcc30 0x7fb7880fd040 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:31.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.075+0000 7fb78ea00700 1 -- 172.21.15.27:0/803646749 >> 172.21.15.27:0/803646749 conn(0x7fb7880f83d0 msgr2=0x7fb7880fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:31.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.075+0000 7fb78ea00700 1 -- 172.21.15.27:0/803646749 shutdown_connections 2024-04-16T05:41:31.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.076+0000 7fb78ea00700 1 -- 172.21.15.27:0/803646749 wait complete. 2024-04-16T05:41:31.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.077+0000 7fb78ea00700 1 Processor -- start 2024-04-16T05:41:31.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.077+0000 7fb78ea00700 1 -- start start 2024-04-16T05:41:31.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.077+0000 7fb78ea00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7880fcc30 0x7fb78810fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:31.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.077+0000 7fb78ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7880fdee0 0x7fb78810ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:31.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.077+0000 7fb78ea00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb788110650 con 0x7fb7880fdee0 2024-04-16T05:41:31.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.077+0000 7fb78ea00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb788110e40 con 0x7fb7880fcc30 2024-04-16T05:41:31.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.078+0000 7fb78cc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7880fdee0 0x7fb78810ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:31.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.078+0000 7fb78cc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7880fdee0 0x7fb78810ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53896/0 (socket says 172.21.15.27:53896) 2024-04-16T05:41:31.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.078+0000 7fb78cc00700 1 -- 172.21.15.27:0/2476226419 learned_addr learned my addr 172.21.15.27:0/2476226419 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:41:31.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.078+0000 7fb78d600700 1 --2- 172.21.15.27:0/2476226419 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7880fcc30 0x7fb78810fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:31.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.078+0000 7fb78cc00700 1 -- 172.21.15.27:0/2476226419 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7880fcc30 msgr2=0x7fb78810fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:31.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.078+0000 7fb78cc00700 1 --2- 172.21.15.27:0/2476226419 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7880fcc30 0x7fb78810fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:31.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.078+0000 7fb78cc00700 1 -- 172.21.15.27:0/2476226419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb77c005040 con 0x7fb7880fdee0 2024-04-16T05:41:31.077 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.079+0000 7fb78cc00700 1 --2- 172.21.15.27:0/2476226419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7880fdee0 0x7fb78810ffc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb77400b790 tx=0x7fb774009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:31.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.079+0000 7fb785600700 1 -- 172.21.15.27:0/2476226419 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb77400f770 con 0x7fb7880fdee0 2024-04-16T05:41:31.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.079+0000 7fb785600700 1 -- 172.21.15.27:0/2476226419 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb77400fda0 con 0x7fb7880fdee0 2024-04-16T05:41:31.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.079+0000 7fb78ea00700 1 -- 172.21.15.27:0/2476226419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb788111120 con 0x7fb7880fdee0 2024-04-16T05:41:31.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.079+0000 7fb785600700 1 -- 172.21.15.27:0/2476226419 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb77400d310 con 0x7fb7880fdee0 2024-04-16T05:41:31.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.079+0000 7fb78ea00700 1 -- 172.21.15.27:0/2476226419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7881116e0 con 0x7fb7880fdee0 2024-04-16T05:41:31.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.081+0000 7fb78ea00700 1 -- 172.21.15.27:0/2476226419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb788101870 con 0x7fb7880fdee0 2024-04-16T05:41:31.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.082+0000 7fb785600700 1 -- 172.21.15.27:0/2476226419 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fb774010420 con 0x7fb7880fdee0 2024-04-16T05:41:31.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.084+0000 7fb785600700 1 --2- 172.21.15.27:0/2476226419 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb7700692a0 0x7fb77006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:31.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.084+0000 7fb785600700 1 -- 172.21.15.27:0/2476226419 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fb7740872b0 con 0x7fb7880fdee0 2024-04-16T05:41:31.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.084+0000 7fb78d600700 1 --2- 172.21.15.27:0/2476226419 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb7700692a0 0x7fb77006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:31.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.085+0000 7fb78d600700 1 --2- 172.21.15.27:0/2476226419 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb7700692a0 0x7fb77006b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00b3b0 tx=0x7fb77c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:31.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.088+0000 7fb785600700 1 -- 172.21.15.27:0/2476226419 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb774056f80 con 0x7fb7880fdee0 2024-04-16T05:41:31.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.341+0000 7fb78ea00700 1 -- 172.21.15.27:0/2476226419 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb788111960 con 0x7fb7700692a0 2024-04-16T05:41:31.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.343+0000 7fb785600700 1 -- 172.21.15.27:0/2476226419 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7fb788111960 con 0x7fb7700692a0 2024-04-16T05:41:31.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.346+0000 7fb78ea00700 1 -- 172.21.15.27:0/2476226419 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb7700692a0 msgr2=0x7fb77006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:31.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.346+0000 7fb78ea00700 1 --2- 172.21.15.27:0/2476226419 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb7700692a0 0x7fb77006b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00b3b0 tx=0x7fb77c009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:31.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.346+0000 7fb78ea00700 1 -- 172.21.15.27:0/2476226419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7880fdee0 msgr2=0x7fb78810ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:31.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.346+0000 7fb78ea00700 1 --2- 172.21.15.27:0/2476226419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7880fdee0 0x7fb78810ffc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb77400b790 tx=0x7fb774009250 comp rx=0 tx=0).stop 2024-04-16T05:41:31.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.346+0000 7fb78ea00700 1 -- 172.21.15.27:0/2476226419 shutdown_connections 2024-04-16T05:41:31.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.346+0000 7fb78ea00700 1 --2- 172.21.15.27:0/2476226419 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb7700692a0 0x7fb77006b750 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:31.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.346+0000 7fb78ea00700 1 --2- 172.21.15.27:0/2476226419 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb7880fcc30 0x7fb78810fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:31.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.346+0000 7fb78ea00700 1 --2- 172.21.15.27:0/2476226419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7880fdee0 0x7fb78810ffc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:31.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.346+0000 7fb78ea00700 1 -- 172.21.15.27:0/2476226419 >> 172.21.15.27:0/2476226419 conn(0x7fb7880f83d0 msgr2=0x7fb7880f99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:31.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.347+0000 7fb78ea00700 1 -- 172.21.15.27:0/2476226419 shutdown_connections 2024-04-16T05:41:31.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.347+0000 7fb78ea00700 1 -- 172.21.15.27:0/2476226419 wait complete. 2024-04-16T05:41:31.361 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:41:31.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.515+0000 7f8405200700 1 -- 172.21.15.27:0/3275594875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84000fcc30 msgr2=0x7f84000fd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:31.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.515+0000 7f8405200700 1 --2- 172.21.15.27:0/3275594875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84000fcc30 0x7f84000fd040 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f83f0004940 tx=0x7f83f0009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:31.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.515+0000 7f8405200700 1 -- 172.21.15.27:0/3275594875 shutdown_connections 2024-04-16T05:41:31.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.515+0000 7f8405200700 1 --2- 172.21.15.27:0/3275594875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84000fdee0 0x7f84000fe350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:31.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.515+0000 7f8405200700 1 --2- 172.21.15.27:0/3275594875 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84000fcc30 0x7f84000fd040 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:31.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.515+0000 7f8405200700 1 -- 172.21.15.27:0/3275594875 >> 172.21.15.27:0/3275594875 conn(0x7f84000f83d0 msgr2=0x7f84000fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:31.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.515+0000 7f8405200700 1 -- 172.21.15.27:0/3275594875 shutdown_connections 2024-04-16T05:41:31.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.515+0000 7f8405200700 1 -- 172.21.15.27:0/3275594875 wait complete. 2024-04-16T05:41:31.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.517+0000 7f8405200700 1 Processor -- start 2024-04-16T05:41:31.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.517+0000 7f8405200700 1 -- start start 2024-04-16T05:41:31.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.518+0000 7f8405200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84000fcc30 0x7f8400071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:31.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.518+0000 7f8405200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84000fdee0 0x7f8400070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:31.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.518+0000 7f8405200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84000708c0 con 0x7f84000fdee0 2024-04-16T05:41:31.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.518+0000 7f8405200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8400070a30 con 0x7f84000fcc30 2024-04-16T05:41:31.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.518+0000 7f83fea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84000fdee0 0x7f8400070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:31.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.518+0000 7f83fea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84000fdee0 0x7f8400070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53920/0 (socket says 172.21.15.27:53920) 2024-04-16T05:41:31.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.518+0000 7f83fea00700 1 -- 172.21.15.27:0/3946024237 learned_addr learned my addr 172.21.15.27:0/3946024237 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:41:31.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.518+0000 7f83ff400700 1 --2- 172.21.15.27:0/3946024237 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84000fcc30 0x7f8400071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:31.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.519+0000 7f83fea00700 1 -- 172.21.15.27:0/3946024237 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84000fcc30 msgr2=0x7f8400071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:31.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.519+0000 7f83fea00700 1 --2- 172.21.15.27:0/3946024237 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84000fcc30 0x7f8400071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:31.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.519+0000 7f83fea00700 1 -- 172.21.15.27:0/3946024237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f0005040 con 0x7f84000fdee0 2024-04-16T05:41:31.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.519+0000 7f83fea00700 1 --2- 172.21.15.27:0/3946024237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84000fdee0 0x7f8400070310 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f83ec002a50 tx=0x7f83ec0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:31.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.520+0000 7f83f7400700 1 -- 172.21.15.27:0/3946024237 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83ec007cc0 con 0x7f84000fdee0 2024-04-16T05:41:31.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.520+0000 7f83f7400700 1 -- 172.21.15.27:0/3946024237 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83ec00f040 con 0x7f84000fdee0 2024-04-16T05:41:31.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.520+0000 7f83f7400700 1 -- 172.21.15.27:0/3946024237 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f83ec00da30 con 0x7f84000fdee0 2024-04-16T05:41:31.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.520+0000 7f8405200700 1 -- 172.21.15.27:0/3946024237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8400070d10 con 0x7f84000fdee0 2024-04-16T05:41:31.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.520+0000 7f8405200700 1 -- 172.21.15.27:0/3946024237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8400115850 con 0x7f84000fdee0 2024-04-16T05:41:31.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.521+0000 7f8405200700 1 -- 172.21.15.27:0/3946024237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f840005f5f0 con 0x7f84000fdee0 2024-04-16T05:41:31.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.523+0000 7f83f7400700 1 -- 172.21.15.27:0/3946024237 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f83ec00dbb0 con 0x7f84000fdee0 2024-04-16T05:41:31.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.524+0000 7f83f7400700 1 --2- 172.21.15.27:0/3946024237 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f83e40692b0 0x7f83e406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:31.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.524+0000 7f83f7400700 1 -- 172.21.15.27:0/3946024237 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f83ec087580 con 0x7f84000fdee0 2024-04-16T05:41:31.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.525+0000 7f83ff400700 1 --2- 172.21.15.27:0/3946024237 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f83e40692b0 0x7f83e406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:31.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.525+0000 7f83ff400700 1 --2- 172.21.15.27:0/3946024237 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f83e40692b0 0x7f83e406b760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f83f0000c00 tx=0x7f83f0009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:31.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.527+0000 7f83f7400700 1 -- 172.21.15.27:0/3946024237 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f83ec053840 con 0x7f84000fdee0 2024-04-16T05:41:31.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.782+0000 7f8405200700 1 -- 172.21.15.27:0/3946024237 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8400102200 con 0x7f83e40692b0 2024-04-16T05:41:31.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.785+0000 7f83f7400700 1 -- 172.21.15.27:0/3946024237 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f8400102200 con 0x7f83e40692b0 2024-04-16T05:41:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.789+0000 7f8405200700 1 -- 172.21.15.27:0/3946024237 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f83e40692b0 msgr2=0x7f83e406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.789+0000 7f8405200700 1 --2- 172.21.15.27:0/3946024237 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f83e40692b0 0x7f83e406b760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f83f0000c00 tx=0x7f83f0009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:31.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.789+0000 7f8405200700 1 -- 172.21.15.27:0/3946024237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84000fdee0 msgr2=0x7f8400070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:31.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.789+0000 7f8405200700 1 --2- 172.21.15.27:0/3946024237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84000fdee0 0x7f8400070310 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f83ec002a50 tx=0x7f83ec0046b0 comp rx=0 tx=0).stop 2024-04-16T05:41:31.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.789+0000 7f8405200700 1 -- 172.21.15.27:0/3946024237 shutdown_connections 2024-04-16T05:41:31.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.789+0000 7f8405200700 1 --2- 172.21.15.27:0/3946024237 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f83e40692b0 0x7f83e406b760 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:31.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.789+0000 7f8405200700 1 --2- 172.21.15.27:0/3946024237 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f84000fcc30 0x7f8400071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:31.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.789+0000 7f8405200700 1 --2- 172.21.15.27:0/3946024237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f84000fdee0 0x7f8400070310 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:31.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.789+0000 7f8405200700 1 -- 172.21.15.27:0/3946024237 >> 172.21.15.27:0/3946024237 conn(0x7f84000f83d0 msgr2=0x7f84000f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:31.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.789+0000 7f8405200700 1 -- 172.21.15.27:0/3946024237 shutdown_connections 2024-04-16T05:41:31.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.789+0000 7f8405200700 1 -- 172.21.15.27:0/3946024237 wait complete. 2024-04-16T05:41:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.963+0000 7f1542800700 1 -- 172.21.15.27:0/2850015197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0662f0 msgr2=0x7f153c0ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.963+0000 7f1542800700 1 --2- 172.21.15.27:0/2850015197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0662f0 0x7f153c0ffa30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f152c004940 tx=0x7f152c009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.963+0000 7f1542800700 1 -- 172.21.15.27:0/2850015197 shutdown_connections 2024-04-16T05:41:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.963+0000 7f1542800700 1 --2- 172.21.15.27:0/2850015197 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f153c0fff70 0x7f153c102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.963+0000 7f1542800700 1 --2- 172.21.15.27:0/2850015197 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0662f0 0x7f153c0ffa30 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:31.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.963+0000 7f1542800700 1 -- 172.21.15.27:0/2850015197 >> 172.21.15.27:0/2850015197 conn(0x7f153c0f74c0 msgr2=0x7f153c0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.963+0000 7f1542800700 1 -- 172.21.15.27:0/2850015197 shutdown_connections 2024-04-16T05:41:31.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.963+0000 7f1542800700 1 -- 172.21.15.27:0/2850015197 wait complete. 2024-04-16T05:41:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.964+0000 7f1542800700 1 Processor -- start 2024-04-16T05:41:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.965+0000 7f1542800700 1 -- start start 2024-04-16T05:41:31.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.965+0000 7f1542800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f153c0662f0 0x7f153c10fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.965+0000 7f1542800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0fff70 0x7f153c110190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.965+0000 7f1542800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c110790 con 0x7f153c0fff70 2024-04-16T05:41:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.965+0000 7f1542800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c110fe0 con 0x7f153c0662f0 2024-04-16T05:41:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.965+0000 7f153aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0fff70 0x7f153c110190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.965+0000 7f153aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0fff70 0x7f153c110190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53938/0 (socket says 172.21.15.27:53938) 2024-04-16T05:41:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.965+0000 7f153aa00700 1 -- 172.21.15.27:0/4033548863 learned_addr learned my addr 172.21.15.27:0/4033548863 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:41:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.966+0000 7f153b400700 1 --2- 172.21.15.27:0/4033548863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f153c0662f0 0x7f153c10fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.966+0000 7f153aa00700 1 -- 172.21.15.27:0/4033548863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f153c0662f0 msgr2=0x7f153c10fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:31.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.966+0000 7f153aa00700 1 --2- 172.21.15.27:0/4033548863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f153c0662f0 0x7f153c10fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:31.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.966+0000 7f153aa00700 1 -- 172.21.15.27:0/4033548863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152c005040 con 0x7f153c0fff70 2024-04-16T05:41:31.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.966+0000 7f153aa00700 1 --2- 172.21.15.27:0/4033548863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0fff70 0x7f153c110190 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f152800c830 tx=0x7f1528004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:31.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.967+0000 7f1533400700 1 -- 172.21.15.27:0/4033548863 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15280106f0 con 0x7f153c0fff70 2024-04-16T05:41:31.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.967+0000 7f1533400700 1 -- 172.21.15.27:0/4033548863 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1528010d20 con 0x7f153c0fff70 2024-04-16T05:41:31.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.967+0000 7f1542800700 1 -- 172.21.15.27:0/4033548863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f153c1112c0 con 0x7f153c0fff70 2024-04-16T05:41:31.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.967+0000 7f1533400700 1 -- 172.21.15.27:0/4033548863 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15280113a0 con 0x7f153c0fff70 2024-04-16T05:41:31.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.967+0000 7f1542800700 1 -- 172.21.15.27:0/4033548863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f153c111800 con 0x7f153c0fff70 2024-04-16T05:41:31.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.968+0000 7f1542800700 1 -- 172.21.15.27:0/4033548863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153c0f8ff0 con 0x7f153c0fff70 2024-04-16T05:41:31.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.969+0000 7f1533400700 1 -- 172.21.15.27:0/4033548863 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f1528012400 con 0x7f153c0fff70 2024-04-16T05:41:31.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.970+0000 7f1533400700 1 --2- 172.21.15.27:0/4033548863 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f15200692b0 0x7f152006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:31.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.971+0000 7f1533400700 1 -- 172.21.15.27:0/4033548863 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1528087250 con 0x7f153c0fff70 2024-04-16T05:41:31.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.971+0000 7f153b400700 1 --2- 172.21.15.27:0/4033548863 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f15200692b0 0x7f152006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:31.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.972+0000 7f153b400700 1 --2- 172.21.15.27:0/4033548863 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f15200692b0 0x7f152006b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f152c00b3b0 tx=0x7f152c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:31.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:31.978+0000 7f1533400700 1 -- 172.21.15.27:0/4033548863 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f1528056f20 con 0x7f153c0fff70 2024-04-16T05:41:32.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.249+0000 7f1542800700 1 -- 172.21.15.27:0/4033548863 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f153c111a80 con 0x7f15200692b0 2024-04-16T05:41:32.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.275+0000 7f1533400700 1 -- 172.21.15.27:0/4033548863 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f153c111a80 con 0x7f15200692b0 2024-04-16T05:41:32.274 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:41:32.274 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (6m) 42s ago 8m 13.4M - 0.25.0 c8568f914cd2 95a43269c2c6 2024-04-16T05:41:32.274 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (8m) 42s ago 8m 6665k - 17.2.7-847-g45317f95 b333581ccdf5 f1acb6959c63 2024-04-16T05:41:32.274 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (7m) 47s ago 7m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 e81efc20e8c0 2024-04-16T05:41:32.274 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (6m) 42s ago 8m 73.3M - 9.4.7 954c08fa6188 0de7ee73579d 2024-04-16T05:41:32.274 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (67s) 42s ago 66s 12.8M - 17.2.7-847-g45317f95 b333581ccdf5 0824e2621be5 2024-04-16T05:41:32.274 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (76s) 42s ago 75s 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 b332a72cc594 2024-04-16T05:41:32.274 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (72s) 47s ago 71s 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 f5e36815ddfa 2024-04-16T05:41:32.274 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (63s) 47s ago 63s 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 755a73c65a7f 2024-04-16T05:41:32.275 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:9283 running (10m) 42s ago 10m 481M - 17.2.7-847-g45317f95 b333581ccdf5 78f461c198fe 2024-04-16T05:41:32.275 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283 running (7m) 47s ago 7m 434M - 17.2.7-847-g45317f95 b333581ccdf5 1ed822ae132b 2024-04-16T05:41:32.275 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (10m) 42s ago 10m 44.8M 2048M 17.2.7-847-g45317f95 b333581ccdf5 88575dbb8d98 2024-04-16T05:41:32.275 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (7m) 47s ago 7m 32.4M 2048M 17.2.7-847-g45317f95 b333581ccdf5 5df19eab6d21 2024-04-16T05:41:32.275 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (8m) 42s ago 8m 7763k - 1.5.0 0da6a335fe13 26bcc8d81c89 2024-04-16T05:41:32.275 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (7m) 47s ago 7m 7717k - 1.5.0 0da6a335fe13 7f796c66dff6 2024-04-16T05:41:32.275 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (6m) 42s ago 6m 44.8M 4288M 17.2.7-847-g45317f95 b333581ccdf5 15e049573bca 2024-04-16T05:41:32.275 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (5m) 42s ago 5m 45.1M 4288M 17.2.7-847-g45317f95 b333581ccdf5 e146295f6500 2024-04-16T05:41:32.275 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (4m) 42s ago 4m 46.0M 4288M 17.2.7-847-g45317f95 b333581ccdf5 6212f6a18fa4 2024-04-16T05:41:32.275 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (4m) 47s ago 4m 42.9M 5312M 17.2.7-847-g45317f95 b333581ccdf5 d6724ad05f9f 2024-04-16T05:41:32.275 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (3m) 47s ago 3m 41.1M 5312M 17.2.7-847-g45317f95 b333581ccdf5 7a59908a3e10 2024-04-16T05:41:32.275 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (3m) 47s ago 3m 40.5M 5312M 17.2.7-847-g45317f95 b333581ccdf5 8e4b60163907 2024-04-16T05:41:32.275 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (6m) 42s ago 7m 46.8M - 2.43.0 a07b618ecd1d 719577e1340d 2024-04-16T05:41:32.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.278+0000 7f1542800700 1 -- 172.21.15.27:0/4033548863 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f15200692b0 msgr2=0x7f152006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:32.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.278+0000 7f1542800700 1 --2- 172.21.15.27:0/4033548863 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f15200692b0 0x7f152006b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f152c00b3b0 tx=0x7f152c009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:32.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.278+0000 7f1542800700 1 -- 172.21.15.27:0/4033548863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0fff70 msgr2=0x7f153c110190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:32.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.278+0000 7f1542800700 1 --2- 172.21.15.27:0/4033548863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0fff70 0x7f153c110190 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f152800c830 tx=0x7f1528004470 comp rx=0 tx=0).stop 2024-04-16T05:41:32.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.278+0000 7f1542800700 1 -- 172.21.15.27:0/4033548863 shutdown_connections 2024-04-16T05:41:32.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.278+0000 7f1542800700 1 --2- 172.21.15.27:0/4033548863 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f15200692b0 0x7f152006b760 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:32.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.278+0000 7f1542800700 1 --2- 172.21.15.27:0/4033548863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f153c0662f0 0x7f153c10fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:32.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.278+0000 7f1542800700 1 --2- 172.21.15.27:0/4033548863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f153c0fff70 0x7f153c110190 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:32.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.278+0000 7f1542800700 1 -- 172.21.15.27:0/4033548863 >> 172.21.15.27:0/4033548863 conn(0x7f153c0f74c0 msgr2=0x7f153c0fbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:32.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.279+0000 7f1542800700 1 -- 172.21.15.27:0/4033548863 shutdown_connections 2024-04-16T05:41:32.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.279+0000 7f1542800700 1 -- 172.21.15.27:0/4033548863 wait complete. 2024-04-16T05:41:32.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.441+0000 7fc0c9a00700 1 -- 172.21.15.27:0/2697752232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0c4071e40 msgr2=0x7fc0c4070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:32.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.441+0000 7fc0c9a00700 1 --2- 172.21.15.27:0/2697752232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0c4071e40 0x7fc0c4070350 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4004940 tx=0x7fc0b4009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:32.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.441+0000 7fc0c9a00700 1 -- 172.21.15.27:0/2697752232 shutdown_connections 2024-04-16T05:41:32.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.441+0000 7fc0c9a00700 1 --2- 172.21.15.27:0/2697752232 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0c4070890 0x7fc0c4070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:32.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.441+0000 7fc0c9a00700 1 --2- 172.21.15.27:0/2697752232 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0c4071e40 0x7fc0c4070350 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:32.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.441+0000 7fc0c9a00700 1 -- 172.21.15.27:0/2697752232 >> 172.21.15.27:0/2697752232 conn(0x7fc0c40f8530 msgr2=0x7fc0c40fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:32.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.441+0000 7fc0c9a00700 1 -- 172.21.15.27:0/2697752232 shutdown_connections 2024-04-16T05:41:32.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.441+0000 7fc0c9a00700 1 -- 172.21.15.27:0/2697752232 wait complete. 2024-04-16T05:41:32.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.442+0000 7fc0c9a00700 1 Processor -- start 2024-04-16T05:41:32.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.443+0000 7fc0c9a00700 1 -- start start 2024-04-16T05:41:32.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.443+0000 7fc0c9a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0c4070890 0x7fc0c410fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:32.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.443+0000 7fc0c9a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0c4071e40 0x7fc0c41101a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:32.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.443+0000 7fc0c9a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c41107a0 con 0x7fc0c4070890 2024-04-16T05:41:32.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.443+0000 7fc0c9a00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c4110ff0 con 0x7fc0c4071e40 2024-04-16T05:41:32.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.443+0000 7fc0c2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0c4070890 0x7fc0c410fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:32.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.443+0000 7fc0c2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0c4070890 0x7fc0c410fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53946/0 (socket says 172.21.15.27:53946) 2024-04-16T05:41:32.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.443+0000 7fc0c2a00700 1 -- 172.21.15.27:0/4046407912 learned_addr learned my addr 172.21.15.27:0/4046407912 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:41:32.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.444+0000 7fc0c2000700 1 --2- 172.21.15.27:0/4046407912 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0c4071e40 0x7fc0c41101a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:32.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.444+0000 7fc0c2a00700 1 -- 172.21.15.27:0/4046407912 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0c4071e40 msgr2=0x7fc0c41101a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:32.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.444+0000 7fc0c2a00700 1 --2- 172.21.15.27:0/4046407912 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0c4071e40 0x7fc0c41101a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:32.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.444+0000 7fc0c2a00700 1 -- 172.21.15.27:0/4046407912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b4005040 con 0x7fc0c4070890 2024-04-16T05:41:32.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.445+0000 7fc0c2a00700 1 --2- 172.21.15.27:0/4046407912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0c4070890 0x7fc0c410fc60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc0b400b970 tx=0x7fc0b4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:32.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.445+0000 7fc0baa00700 1 -- 172.21.15.27:0/4046407912 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0b400e040 con 0x7fc0c4070890 2024-04-16T05:41:32.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.445+0000 7fc0baa00700 1 -- 172.21.15.27:0/4046407912 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc0b400bbf0 con 0x7fc0c4070890 2024-04-16T05:41:32.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.445+0000 7fc0c9a00700 1 -- 172.21.15.27:0/4046407912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c4111270 con 0x7fc0c4070890 2024-04-16T05:41:32.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.445+0000 7fc0baa00700 1 -- 172.21.15.27:0/4046407912 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0b401c3f0 con 0x7fc0c4070890 2024-04-16T05:41:32.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.445+0000 7fc0c9a00700 1 -- 172.21.15.27:0/4046407912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c41117b0 con 0x7fc0c4070890 2024-04-16T05:41:32.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.446+0000 7fc0c9a00700 1 -- 172.21.15.27:0/4046407912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0c405f5f0 con 0x7fc0c4070890 2024-04-16T05:41:32.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.448+0000 7fc0baa00700 1 -- 172.21.15.27:0/4046407912 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fc0b400bd60 con 0x7fc0c4070890 2024-04-16T05:41:32.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.449+0000 7fc0baa00700 1 --2- 172.21.15.27:0/4046407912 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc0a80692b0 0x7fc0a806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:32.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.449+0000 7fc0baa00700 1 -- 172.21.15.27:0/4046407912 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc0b4088530 con 0x7fc0c4070890 2024-04-16T05:41:32.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.449+0000 7fc0c2000700 1 --2- 172.21.15.27:0/4046407912 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc0a80692b0 0x7fc0a806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:32.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.450+0000 7fc0c2000700 1 --2- 172.21.15.27:0/4046407912 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc0a80692b0 0x7fc0a806b760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0004f40 tx=0x7fc0b000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:32.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.453+0000 7fc0baa00700 1 -- 172.21.15.27:0/4046407912 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fc0b4058350 con 0x7fc0c4070890 2024-04-16T05:41:32.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.799+0000 7fc0c9a00700 1 -- 172.21.15.27:0/4046407912 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc0c4111b30 con 0x7fc0c4070890 2024-04-16T05:41:32.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.800+0000 7fc0baa00700 1 -- 172.21.15.27:0/4046407912 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7fc0b4057ee0 con 0x7fc0c4070890 2024-04-16T05:41:32.799 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:41:32.799 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:41:32.799 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T05:41:32.799 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:41:32.799 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:41:32.799 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T05:41:32.799 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:41:32.800 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:41:32.800 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T05:41:32.800 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:41:32.800 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:41:32.800 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T05:41:32.800 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:41:32.800 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:41:32.800 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 14 2024-04-16T05:41:32.800 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:41:32.800 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:41:32.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.804+0000 7fc0c9a00700 1 -- 172.21.15.27:0/4046407912 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc0a80692b0 msgr2=0x7fc0a806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:32.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.804+0000 7fc0c9a00700 1 --2- 172.21.15.27:0/4046407912 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc0a80692b0 0x7fc0a806b760 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0004f40 tx=0x7fc0b000b040 comp rx=0 tx=0).stop 2024-04-16T05:41:32.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.804+0000 7fc0c9a00700 1 -- 172.21.15.27:0/4046407912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0c4070890 msgr2=0x7fc0c410fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.804+0000 7fc0c9a00700 1 --2- 172.21.15.27:0/4046407912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0c4070890 0x7fc0c410fc60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc0b400b970 tx=0x7fc0b4009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.804+0000 7fc0c9a00700 1 -- 172.21.15.27:0/4046407912 shutdown_connections 2024-04-16T05:41:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.804+0000 7fc0c9a00700 1 --2- 172.21.15.27:0/4046407912 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fc0a80692b0 0x7fc0a806b760 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.804+0000 7fc0c9a00700 1 --2- 172.21.15.27:0/4046407912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0c4070890 0x7fc0c410fc60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.804+0000 7fc0c9a00700 1 --2- 172.21.15.27:0/4046407912 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0c4071e40 0x7fc0c41101a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.804+0000 7fc0c9a00700 1 -- 172.21.15.27:0/4046407912 >> 172.21.15.27:0/4046407912 conn(0x7fc0c40f8530 msgr2=0x7fc0c40f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.805+0000 7fc0c9a00700 1 -- 172.21.15.27:0/4046407912 shutdown_connections 2024-04-16T05:41:32.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.805+0000 7fc0c9a00700 1 -- 172.21.15.27:0/4046407912 wait complete. 2024-04-16T05:41:32.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.970+0000 7f9e72c00700 1 -- 172.21.15.27:0/3230123444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e6c0fdba0 msgr2=0x7f9e6c1044f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:32.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.970+0000 7f9e72c00700 1 --2- 172.21.15.27:0/3230123444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e6c0fdba0 0x7f9e6c1044f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9e58004940 tx=0x7f9e58009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:32.969 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:32 smithi027 ceph-mon[29206]: from='client.14548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:41:32.969 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:32 smithi027 ceph-mon[29206]: pgmap v265: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:32.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.970+0000 7f9e72c00700 1 -- 172.21.15.27:0/3230123444 shutdown_connections 2024-04-16T05:41:32.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.970+0000 7f9e72c00700 1 --2- 172.21.15.27:0/3230123444 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e6c0fdba0 0x7f9e6c1044f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:32.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.970+0000 7f9e72c00700 1 --2- 172.21.15.27:0/3230123444 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e6c0fd250 0x7f9e6c0fd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:32.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.970+0000 7f9e72c00700 1 -- 172.21.15.27:0/3230123444 >> 172.21.15.27:0/3230123444 conn(0x7f9e6c0f6a00 msgr2=0x7f9e6c0f8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:32.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.971+0000 7f9e72c00700 1 -- 172.21.15.27:0/3230123444 shutdown_connections 2024-04-16T05:41:32.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.971+0000 7f9e72c00700 1 -- 172.21.15.27:0/3230123444 wait complete. 2024-04-16T05:41:32.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.972+0000 7f9e72c00700 1 Processor -- start 2024-04-16T05:41:32.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.972+0000 7f9e72c00700 1 -- start start 2024-04-16T05:41:32.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.972+0000 7f9e72c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e6c0fd250 0x7f9e6c10fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:32.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.972+0000 7f9e72c00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e6c0fdba0 0x7f9e6c110000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:32.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.972+0000 7f9e72c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c110600 con 0x7f9e6c0fd250 2024-04-16T05:41:32.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.972+0000 7f9e72c00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c110e50 con 0x7f9e6c0fdba0 2024-04-16T05:41:32.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.973+0000 7f9e71800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e6c0fd250 0x7f9e6c10fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:32.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.973+0000 7f9e71800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e6c0fd250 0x7f9e6c10fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53968/0 (socket says 172.21.15.27:53968) 2024-04-16T05:41:32.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.973+0000 7f9e71800700 1 -- 172.21.15.27:0/2790162241 learned_addr learned my addr 172.21.15.27:0/2790162241 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:41:32.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.973+0000 7f9e70e00700 1 --2- 172.21.15.27:0/2790162241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e6c0fdba0 0x7f9e6c110000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:32.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.973+0000 7f9e71800700 1 -- 172.21.15.27:0/2790162241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e6c0fdba0 msgr2=0x7f9e6c110000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:32.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.973+0000 7f9e71800700 1 --2- 172.21.15.27:0/2790162241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e6c0fdba0 0x7f9e6c110000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:32.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.973+0000 7f9e71800700 1 -- 172.21.15.27:0/2790162241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e58005040 con 0x7f9e6c0fd250 2024-04-16T05:41:32.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.974+0000 7f9e71800700 1 --2- 172.21.15.27:0/2790162241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e6c0fd250 0x7f9e6c10fac0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9e6000c830 tx=0x7f9e60004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:32.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.974+0000 7f9e69600700 1 -- 172.21.15.27:0/2790162241 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e6000a760 con 0x7f9e6c0fd250 2024-04-16T05:41:32.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.974+0000 7f9e69600700 1 -- 172.21.15.27:0/2790162241 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e6000ad90 con 0x7f9e6c0fd250 2024-04-16T05:41:32.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.974+0000 7f9e69600700 1 -- 172.21.15.27:0/2790162241 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e600113a0 con 0x7f9e6c0fd250 2024-04-16T05:41:32.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.974+0000 7f9e72c00700 1 -- 172.21.15.27:0/2790162241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e6c111130 con 0x7f9e6c0fd250 2024-04-16T05:41:32.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.974+0000 7f9e72c00700 1 -- 172.21.15.27:0/2790162241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e6c111670 con 0x7f9e6c0fd250 2024-04-16T05:41:32.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.976+0000 7f9e72c00700 1 -- 172.21.15.27:0/2790162241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e6c05f5f0 con 0x7f9e6c0fd250 2024-04-16T05:41:32.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.977+0000 7f9e69600700 1 -- 172.21.15.27:0/2790162241 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f9e60012400 con 0x7f9e6c0fd250 2024-04-16T05:41:32.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.978+0000 7f9e69600700 1 --2- 172.21.15.27:0/2790162241 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f9e540692b0 0x7f9e5406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:32.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.978+0000 7f9e69600700 1 -- 172.21.15.27:0/2790162241 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9e60087330 con 0x7f9e6c0fd250 2024-04-16T05:41:32.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.979+0000 7f9e70e00700 1 --2- 172.21.15.27:0/2790162241 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f9e540692b0 0x7f9e5406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:32.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.979+0000 7f9e70e00700 1 --2- 172.21.15.27:0/2790162241 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f9e540692b0 0x7f9e5406b760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9e58004f90 tx=0x7f9e58009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:32.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:32.985+0000 7f9e69600700 1 -- 172.21.15.27:0/2790162241 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f9e60057000 con 0x7f9e6c0fd250 2024-04-16T05:41:33.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:32 smithi122 ceph-mon[35998]: from='client.14548 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:41:33.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:32 smithi122 ceph-mon[35998]: pgmap v265: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:33.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.262+0000 7f9e72c00700 1 -- 172.21.15.27:0/2790162241 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9e6c111a40 con 0x7f9e6c0fd250 2024-04-16T05:41:33.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.263+0000 7f9e69600700 1 -- 172.21.15.27:0/2790162241 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1810 (secure 0 0 0) 0x7f9e60056b90 con 0x7f9e6c0fd250 2024-04-16T05:41:33.261 INFO:teuthology.orchestra.run.smithi027.stdout:e16 2024-04-16T05:41:33.261 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:41:33.261 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:41:33.261 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:41:33.261 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:41:33.262 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:41:33.262 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:41:33.262 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 15 2024-04-16T05:41:33.262 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:41:33.262 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:41:33.262 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:40:32.701649+0000 2024-04-16T05:41:33.262 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:41:33.262 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:41:33.262 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:41:33.262 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:41:33.262 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:41:33.262 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:41:33.262 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:41:33.262 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 46 2024-04-16T05:41:33.262 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:41:33.263 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:41:33.263 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:41:33.263 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=14462} 2024-04-16T05:41:33.263 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:41:33.263 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:41:33.263 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:41:33.263 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:41:33.263 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:41:33.263 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:41:33.263 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:41:33.263 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:41:33.263 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:41:33.263 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:41:33.263 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:41:33.264 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:41:33.264 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:41:33.264 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{-1:14474} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2065690722,v1:172.21.15.27:6827/2065690722] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:41:33.264 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:24263} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:41:33.264 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:24273} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:41:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.266+0000 7f9e72c00700 1 -- 172.21.15.27:0/2790162241 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f9e540692b0 msgr2=0x7f9e5406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.266+0000 7f9e72c00700 1 --2- 172.21.15.27:0/2790162241 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f9e540692b0 0x7f9e5406b760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9e58004f90 tx=0x7f9e58009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.266+0000 7f9e72c00700 1 -- 172.21.15.27:0/2790162241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e6c0fd250 msgr2=0x7f9e6c10fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:33.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.266+0000 7f9e72c00700 1 --2- 172.21.15.27:0/2790162241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e6c0fd250 0x7f9e6c10fac0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9e6000c830 tx=0x7f9e60004470 comp rx=0 tx=0).stop 2024-04-16T05:41:33.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.266+0000 7f9e72c00700 1 -- 172.21.15.27:0/2790162241 shutdown_connections 2024-04-16T05:41:33.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.266+0000 7f9e72c00700 1 --2- 172.21.15.27:0/2790162241 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f9e540692b0 0x7f9e5406b760 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:33.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.266+0000 7f9e72c00700 1 --2- 172.21.15.27:0/2790162241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e6c0fd250 0x7f9e6c10fac0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:33.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.266+0000 7f9e72c00700 1 --2- 172.21.15.27:0/2790162241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e6c0fdba0 0x7f9e6c110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:33.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.266+0000 7f9e72c00700 1 -- 172.21.15.27:0/2790162241 >> 172.21.15.27:0/2790162241 conn(0x7f9e6c0f6a00 msgr2=0x7f9e6c0f6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:33.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.267+0000 7f9e72c00700 1 -- 172.21.15.27:0/2790162241 shutdown_connections 2024-04-16T05:41:33.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.267+0000 7f9e72c00700 1 -- 172.21.15.27:0/2790162241 wait complete. 2024-04-16T05:41:33.266 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2024-04-16T05:41:33.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.439+0000 7f488e000700 1 -- 172.21.15.27:0/1634411416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48880fcdc0 msgr2=0x7f48880fd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:33.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.439+0000 7f488e000700 1 --2- 172.21.15.27:0/1634411416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48880fcdc0 0x7f48880fd1d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4878004940 tx=0x7f4878009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:33.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.439+0000 7f488e000700 1 -- 172.21.15.27:0/1634411416 shutdown_connections 2024-04-16T05:41:33.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.439+0000 7f488e000700 1 --2- 172.21.15.27:0/1634411416 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48880fd710 0x7f48880fe070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:33.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.439+0000 7f488e000700 1 --2- 172.21.15.27:0/1634411416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48880fcdc0 0x7f48880fd1d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:33.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.439+0000 7f488e000700 1 -- 172.21.15.27:0/1634411416 >> 172.21.15.27:0/1634411416 conn(0x7f48880f8540 msgr2=0x7f48880fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:33.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.439+0000 7f488e000700 1 -- 172.21.15.27:0/1634411416 shutdown_connections 2024-04-16T05:41:33.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.439+0000 7f488e000700 1 -- 172.21.15.27:0/1634411416 wait complete. 2024-04-16T05:41:33.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.440+0000 7f488e000700 1 Processor -- start 2024-04-16T05:41:33.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.440+0000 7f488e000700 1 -- start start 2024-04-16T05:41:33.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.441+0000 7f488e000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48880fcdc0 0x7f488810daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:33.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.441+0000 7f488e000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48880fd710 0x7f488810dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:33.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.441+0000 7f488e000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488810e5e0 con 0x7f48880fd710 2024-04-16T05:41:33.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.441+0000 7f488e000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488810ee30 con 0x7f48880fcdc0 2024-04-16T05:41:33.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.441+0000 7f4886a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48880fd710 0x7f488810dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:33.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.441+0000 7f4886a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48880fd710 0x7f488810dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53986/0 (socket says 172.21.15.27:53986) 2024-04-16T05:41:33.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.441+0000 7f4886a00700 1 -- 172.21.15.27:0/2259772460 learned_addr learned my addr 172.21.15.27:0/2259772460 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:41:33.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.441+0000 7f4887400700 1 --2- 172.21.15.27:0/2259772460 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48880fcdc0 0x7f488810daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:33.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.442+0000 7f4886a00700 1 -- 172.21.15.27:0/2259772460 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48880fcdc0 msgr2=0x7f488810daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:33.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.442+0000 7f4886a00700 1 --2- 172.21.15.27:0/2259772460 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48880fcdc0 0x7f488810daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:33.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.442+0000 7f4886a00700 1 -- 172.21.15.27:0/2259772460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4878005040 con 0x7f48880fd710 2024-04-16T05:41:33.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.442+0000 7f4886a00700 1 --2- 172.21.15.27:0/2259772460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48880fd710 0x7f488810dfe0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f487400c890 tx=0x7f48740044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:33.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.442+0000 7f487f400700 1 -- 172.21.15.27:0/2259772460 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f487400a7d0 con 0x7f48880fd710 2024-04-16T05:41:33.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.442+0000 7f487f400700 1 -- 172.21.15.27:0/2259772460 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f487400ae00 con 0x7f48880fd710 2024-04-16T05:41:33.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.442+0000 7f487f400700 1 -- 172.21.15.27:0/2259772460 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48740113a0 con 0x7f48880fd710 2024-04-16T05:41:33.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.442+0000 7f488e000700 1 -- 172.21.15.27:0/2259772460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f488810f110 con 0x7f48880fd710 2024-04-16T05:41:33.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.443+0000 7f488e000700 1 -- 172.21.15.27:0/2259772460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f488810f650 con 0x7f48880fd710 2024-04-16T05:41:33.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.444+0000 7f488e000700 1 -- 172.21.15.27:0/2259772460 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488805f5f0 con 0x7f48880fd710 2024-04-16T05:41:33.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.445+0000 7f487f400700 1 -- 172.21.15.27:0/2259772460 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f4874012400 con 0x7f48880fd710 2024-04-16T05:41:33.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.446+0000 7f487f400700 1 --2- 172.21.15.27:0/2259772460 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f486c0692b0 0x7f486c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:33.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.446+0000 7f487f400700 1 -- 172.21.15.27:0/2259772460 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4874087350 con 0x7f48880fd710 2024-04-16T05:41:33.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.446+0000 7f4887400700 1 --2- 172.21.15.27:0/2259772460 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f486c0692b0 0x7f486c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:33.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.447+0000 7f4887400700 1 --2- 172.21.15.27:0/2259772460 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f486c0692b0 0x7f486c06b760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4878009fd0 tx=0x7f4878009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:33.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.452+0000 7f487f400700 1 -- 172.21.15.27:0/2259772460 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f4874057020 con 0x7f48880fd710 2024-04-16T05:41:33.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.696+0000 7f488e000700 1 -- 172.21.15.27:0/2259772460 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f488810f970 con 0x7f486c0692b0 2024-04-16T05:41:33.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.697+0000 7f487f400700 1 -- 172.21.15.27:0/2259772460 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f488810f970 con 0x7f486c0692b0 2024-04-16T05:41:33.696 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:41:33.696 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838", 2024-04-16T05:41:33.696 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:41:33.696 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:41:33.696 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [], 2024-04-16T05:41:33.696 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "", 2024-04-16T05:41:33.696 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 image", 2024-04-16T05:41:33.696 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:41:33.697 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:41:33.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.700+0000 7f488e000700 1 -- 172.21.15.27:0/2259772460 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f486c0692b0 msgr2=0x7f486c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:33.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.700+0000 7f488e000700 1 --2- 172.21.15.27:0/2259772460 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f486c0692b0 0x7f486c06b760 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4878009fd0 tx=0x7f4878009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:33.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.700+0000 7f488e000700 1 -- 172.21.15.27:0/2259772460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48880fd710 msgr2=0x7f488810dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:33.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.700+0000 7f488e000700 1 --2- 172.21.15.27:0/2259772460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48880fd710 0x7f488810dfe0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f487400c890 tx=0x7f48740044c0 comp rx=0 tx=0).stop 2024-04-16T05:41:33.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.701+0000 7f488e000700 1 -- 172.21.15.27:0/2259772460 shutdown_connections 2024-04-16T05:41:33.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.701+0000 7f488e000700 1 --2- 172.21.15.27:0/2259772460 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f486c0692b0 0x7f486c06b760 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:33.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.701+0000 7f488e000700 1 --2- 172.21.15.27:0/2259772460 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48880fcdc0 0x7f488810daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:33.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.701+0000 7f488e000700 1 --2- 172.21.15.27:0/2259772460 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48880fd710 0x7f488810dfe0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:33.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.701+0000 7f488e000700 1 -- 172.21.15.27:0/2259772460 >> 172.21.15.27:0/2259772460 conn(0x7f48880f8540 msgr2=0x7f48880f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.701+0000 7f488e000700 1 -- 172.21.15.27:0/2259772460 shutdown_connections 2024-04-16T05:41:33.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.701+0000 7f488e000700 1 -- 172.21.15.27:0/2259772460 wait complete. 2024-04-16T05:41:33.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.863+0000 7f1cbec00700 1 -- 172.21.15.27:0/1106765881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cb80fcef0 msgr2=0x7f1cb80fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:33.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.863+0000 7f1cbec00700 1 --2- 172.21.15.27:0/1106765881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cb80fcef0 0x7f1cb80fd320 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1ca8004940 tx=0x7f1ca8009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:33.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.864+0000 7f1cbec00700 1 -- 172.21.15.27:0/1106765881 shutdown_connections 2024-04-16T05:41:33.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.864+0000 7f1cbec00700 1 --2- 172.21.15.27:0/1106765881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cb80fe0e0 0x7f1cb80fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:33.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.864+0000 7f1cbec00700 1 --2- 172.21.15.27:0/1106765881 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cb80fcef0 0x7f1cb80fd320 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:33.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.864+0000 7f1cbec00700 1 -- 172.21.15.27:0/1106765881 >> 172.21.15.27:0/1106765881 conn(0x7f1cb80f8530 msgr2=0x7f1cb80fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:33.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.864+0000 7f1cbec00700 1 -- 172.21.15.27:0/1106765881 shutdown_connections 2024-04-16T05:41:33.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.864+0000 7f1cbec00700 1 -- 172.21.15.27:0/1106765881 wait complete. 2024-04-16T05:41:33.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.865+0000 7f1cbec00700 1 Processor -- start 2024-04-16T05:41:33.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.865+0000 7f1cbec00700 1 -- start start 2024-04-16T05:41:33.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.865+0000 7f1cbec00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cb80fcef0 0x7f1cb810fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:33.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.865+0000 7f1cbec00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cb80fe0e0 0x7f1cb8110180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:33.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.865+0000 7f1cbec00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb8110810 con 0x7f1cb80fcef0 2024-04-16T05:41:33.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.865+0000 7f1cbec00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb8111000 con 0x7f1cb80fe0e0 2024-04-16T05:41:33.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.866+0000 7f1cb7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cb80fcef0 0x7f1cb810fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:33.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.866+0000 7f1cb7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cb80fcef0 0x7f1cb810fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54006/0 (socket says 172.21.15.27:54006) 2024-04-16T05:41:33.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.866+0000 7f1cb7e00700 1 -- 172.21.15.27:0/611185910 learned_addr learned my addr 172.21.15.27:0/611185910 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:41:33.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.866+0000 7f1cb7e00700 1 -- 172.21.15.27:0/611185910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cb80fe0e0 msgr2=0x7f1cb8110180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:33.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.866+0000 7f1cb7e00700 1 --2- 172.21.15.27:0/611185910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cb80fe0e0 0x7f1cb8110180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:33.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.866+0000 7f1cb7e00700 1 -- 172.21.15.27:0/611185910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca8005040 con 0x7f1cb80fcef0 2024-04-16T05:41:33.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.866+0000 7f1cb7e00700 1 --2- 172.21.15.27:0/611185910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cb80fcef0 0x7f1cb810fc40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1ca800b970 tx=0x7f1ca8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:33.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.867+0000 7f1cafe00700 1 -- 172.21.15.27:0/611185910 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ca800e040 con 0x7f1cb80fcef0 2024-04-16T05:41:33.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.867+0000 7f1cbec00700 1 -- 172.21.15.27:0/611185910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb8111280 con 0x7f1cb80fcef0 2024-04-16T05:41:33.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.867+0000 7f1cafe00700 1 -- 172.21.15.27:0/611185910 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ca8004290 con 0x7f1cb80fcef0 2024-04-16T05:41:33.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.867+0000 7f1cafe00700 1 -- 172.21.15.27:0/611185910 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ca801c630 con 0x7f1cb80fcef0 2024-04-16T05:41:33.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.867+0000 7f1cbec00700 1 -- 172.21.15.27:0/611185910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb8111840 con 0x7f1cb80fcef0 2024-04-16T05:41:33.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.868+0000 7f1cbec00700 1 -- 172.21.15.27:0/611185910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cb8101a70 con 0x7f1cb80fcef0 2024-04-16T05:41:33.868 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.871+0000 7f1cafe00700 1 -- 172.21.15.27:0/611185910 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f1ca800bbf0 con 0x7f1cb80fcef0 2024-04-16T05:41:33.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.871+0000 7f1cafe00700 1 --2- 172.21.15.27:0/611185910 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f1c9c0692b0 0x7f1c9c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:41:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.872+0000 7f1cafe00700 1 -- 172.21.15.27:0/611185910 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1ca8088fc0 con 0x7f1cb80fcef0 2024-04-16T05:41:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.872+0000 7f1cb7400700 1 --2- 172.21.15.27:0/611185910 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f1c9c0692b0 0x7f1c9c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:41:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.873+0000 7f1cb7400700 1 --2- 172.21.15.27:0/611185910 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f1c9c0692b0 0x7f1c9c06b760 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4006fd0 tx=0x7f1ca4006db0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:41:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:33.875+0000 7f1cafe00700 1 -- 172.21.15.27:0/611185910 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f1ca8058dd0 con 0x7f1cb80fcef0 2024-04-16T05:41:34.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:33 smithi122 ceph-mon[35998]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:41:34.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:33 smithi122 ceph-mon[35998]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:41:34.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:33 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/4046407912' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:41:34.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:33 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/2790162241' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:41:34.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:33 smithi027 ceph-mon[29206]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:41:34.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:33 smithi027 ceph-mon[29206]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:41:34.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:33 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/4046407912' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:41:34.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:33 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/2790162241' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:41:34.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:34.192+0000 7f1cbec00700 1 -- 172.21.15.27:0/611185910 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f1cb8105bb0 con 0x7f1cb80fcef0 2024-04-16T05:41:34.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:34.192+0000 7f1cafe00700 1 -- 172.21.15.27:0/611185910 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f1cb8105bb0 con 0x7f1cb80fcef0 2024-04-16T05:41:34.191 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:41:34.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:34.195+0000 7f1cbec00700 1 -- 172.21.15.27:0/611185910 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f1c9c0692b0 msgr2=0x7f1c9c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:34.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:34.195+0000 7f1cbec00700 1 --2- 172.21.15.27:0/611185910 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f1c9c0692b0 0x7f1c9c06b760 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4006fd0 tx=0x7f1ca4006db0 comp rx=0 tx=0).stop 2024-04-16T05:41:34.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:34.195+0000 7f1cbec00700 1 -- 172.21.15.27:0/611185910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cb80fcef0 msgr2=0x7f1cb810fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:41:34.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:34.195+0000 7f1cbec00700 1 --2- 172.21.15.27:0/611185910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cb80fcef0 0x7f1cb810fc40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1ca800b970 tx=0x7f1ca8009e30 comp rx=0 tx=0).stop 2024-04-16T05:41:34.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:34.195+0000 7f1cbec00700 1 -- 172.21.15.27:0/611185910 shutdown_connections 2024-04-16T05:41:34.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:34.195+0000 7f1cbec00700 1 --2- 172.21.15.27:0/611185910 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f1c9c0692b0 0x7f1c9c06b760 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:34.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:34.195+0000 7f1cbec00700 1 --2- 172.21.15.27:0/611185910 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1cb80fcef0 0x7f1cb810fc40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:34.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:34.195+0000 7f1cbec00700 1 --2- 172.21.15.27:0/611185910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cb80fe0e0 0x7f1cb8110180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:41:34.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:34.195+0000 7f1cbec00700 1 -- 172.21.15.27:0/611185910 >> 172.21.15.27:0/611185910 conn(0x7f1cb80f8530 msgr2=0x7f1cb80f9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:41:34.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:34.196+0000 7f1cbec00700 1 -- 172.21.15.27:0/611185910 shutdown_connections 2024-04-16T05:41:34.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:41:34.196+0000 7f1cbec00700 1 -- 172.21.15.27:0/611185910 wait complete. 2024-04-16T05:41:35.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:34 smithi122 ceph-mon[35998]: pgmap v266: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:35.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:34 smithi122 ceph-mon[35998]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:41:35.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:34 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/611185910' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:41:35.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:34 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:41:35.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:34 smithi027 ceph-mon[29206]: pgmap v266: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:35.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:34 smithi027 ceph-mon[29206]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:41:35.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:34 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/611185910' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:41:35.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:34 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:41:37.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:36 smithi122 ceph-mon[35998]: pgmap v267: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:37.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:36 smithi027 ceph-mon[29206]: pgmap v267: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:39.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:38 smithi122 ceph-mon[35998]: pgmap v268: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:39.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:38 smithi027 ceph-mon[29206]: pgmap v268: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:41.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:40 smithi122 ceph-mon[35998]: pgmap v269: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:41.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:40 smithi027 ceph-mon[29206]: pgmap v269: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:43.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:42 smithi122 ceph-mon[35998]: pgmap v270: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:43.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:42 smithi027 ceph-mon[29206]: pgmap v270: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:45.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:44 smithi122 ceph-mon[35998]: pgmap v271: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:45.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:44 smithi027 ceph-mon[29206]: pgmap v271: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:47.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:46 smithi122 ceph-mon[35998]: pgmap v272: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:47.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:46 smithi027 ceph-mon[29206]: pgmap v272: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:49.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:48 smithi122 ceph-mon[35998]: pgmap v273: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:49.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:48 smithi027 ceph-mon[29206]: pgmap v273: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:50.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:49 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:41:50.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:49 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:41:51.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:50 smithi122 ceph-mon[35998]: pgmap v274: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:51.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:50 smithi027 ceph-mon[29206]: pgmap v274: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:52.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:51 smithi122 ceph-mon[35998]: pgmap v275: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:52.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:51 smithi027 ceph-mon[29206]: pgmap v275: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:54.991 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:54 smithi027 ceph-mon[29206]: pgmap v276: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:55.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:54 smithi122 ceph-mon[35998]: pgmap v276: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:57.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:56 smithi122 ceph-mon[35998]: pgmap v277: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:57.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:56 smithi027 ceph-mon[29206]: pgmap v277: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:59.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:41:58 smithi122 ceph-mon[35998]: pgmap v278: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:41:59.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:41:58 smithi027 ceph-mon[29206]: pgmap v278: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:01.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:00 smithi122 ceph-mon[35998]: pgmap v279: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:01.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:00 smithi027 ceph-mon[29206]: pgmap v279: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:03.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:02 smithi122 ceph-mon[35998]: pgmap v280: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:03.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:02 smithi027 ceph-mon[29206]: pgmap v280: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:04.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.365+0000 7fa41b000700 1 -- 172.21.15.27:0/1317923381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4140fff70 msgr2=0x7fa414102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:04.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.365+0000 7fa41b000700 1 --2- 172.21.15.27:0/1317923381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4140fff70 0x7fa414102390 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa400004940 tx=0x7fa400009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:04.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.365+0000 7fa41b000700 1 -- 172.21.15.27:0/1317923381 shutdown_connections 2024-04-16T05:42:04.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.365+0000 7fa41b000700 1 --2- 172.21.15.27:0/1317923381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4140fff70 0x7fa414102390 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:04.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.365+0000 7fa41b000700 1 --2- 172.21.15.27:0/1317923381 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4140662f0 0x7fa4140ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:04.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.365+0000 7fa41b000700 1 -- 172.21.15.27:0/1317923381 >> 172.21.15.27:0/1317923381 conn(0x7fa4140f74c0 msgr2=0x7fa4140f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:04.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.366+0000 7fa41b000700 1 -- 172.21.15.27:0/1317923381 shutdown_connections 2024-04-16T05:42:04.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.366+0000 7fa41b000700 1 -- 172.21.15.27:0/1317923381 wait complete. 2024-04-16T05:42:04.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.367+0000 7fa41b000700 1 Processor -- start 2024-04-16T05:42:04.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.367+0000 7fa41b000700 1 -- start start 2024-04-16T05:42:04.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.368+0000 7fa41b000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4140662f0 0x7fa4140fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.368+0000 7fa41b000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4140fff70 0x7fa4140fb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.368+0000 7fa41b000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4140fbee0 con 0x7fa4140662f0 2024-04-16T05:42:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.368+0000 7fa41b000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4140fc050 con 0x7fa4140fff70 2024-04-16T05:42:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.368+0000 7fa413e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4140662f0 0x7fa4140fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.368+0000 7fa413e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4140662f0 0x7fa4140fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45790/0 (socket says 172.21.15.27:45790) 2024-04-16T05:42:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.368+0000 7fa413e00700 1 -- 172.21.15.27:0/848772725 learned_addr learned my addr 172.21.15.27:0/848772725 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:42:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.368+0000 7fa413400700 1 --2- 172.21.15.27:0/848772725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4140fff70 0x7fa4140fb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.368+0000 7fa413e00700 1 -- 172.21.15.27:0/848772725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4140fff70 msgr2=0x7fa4140fb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.368+0000 7fa413e00700 1 --2- 172.21.15.27:0/848772725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4140fff70 0x7fa4140fb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.368+0000 7fa413e00700 1 -- 172.21.15.27:0/848772725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa400005040 con 0x7fa4140662f0 2024-04-16T05:42:04.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.368+0000 7fa413400700 1 --2- 172.21.15.27:0/848772725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4140fff70 0x7fa4140fb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:42:04.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.369+0000 7fa413e00700 1 --2- 172.21.15.27:0/848772725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4140662f0 0x7fa4140fd460 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa40400c890 tx=0x7fa4040044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:04.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.369+0000 7fa40be00700 1 -- 172.21.15.27:0/848772725 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa40400a7d0 con 0x7fa4140662f0 2024-04-16T05:42:04.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.369+0000 7fa40be00700 1 -- 172.21.15.27:0/848772725 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa40400ae00 con 0x7fa4140662f0 2024-04-16T05:42:04.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.369+0000 7fa40be00700 1 -- 172.21.15.27:0/848772725 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa4040113a0 con 0x7fa4140662f0 2024-04-16T05:42:04.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.369+0000 7fa41b000700 1 -- 172.21.15.27:0/848772725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4140fc330 con 0x7fa4140662f0 2024-04-16T05:42:04.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.369+0000 7fa41b000700 1 -- 172.21.15.27:0/848772725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa414115a90 con 0x7fa4140662f0 2024-04-16T05:42:04.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.371+0000 7fa41b000700 1 -- 172.21.15.27:0/848772725 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa41405f5f0 con 0x7fa4140662f0 2024-04-16T05:42:04.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.372+0000 7fa40be00700 1 -- 172.21.15.27:0/848772725 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fa404012400 con 0x7fa4140662f0 2024-04-16T05:42:04.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.373+0000 7fa40be00700 1 --2- 172.21.15.27:0/848772725 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa3f80692a0 0x7fa3f806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:04.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.373+0000 7fa40be00700 1 -- 172.21.15.27:0/848772725 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa404087370 con 0x7fa4140662f0 2024-04-16T05:42:04.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.373+0000 7fa413400700 1 --2- 172.21.15.27:0/848772725 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa3f80692a0 0x7fa3f806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:04.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.374+0000 7fa413400700 1 --2- 172.21.15.27:0/848772725 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa3f80692a0 0x7fa3f806b750 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa40000b970 tx=0x7fa400009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:04.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.378+0000 7fa40be00700 1 -- 172.21.15.27:0/848772725 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fa4040535b0 con 0x7fa4140662f0 2024-04-16T05:42:04.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.614+0000 7fa41b000700 1 -- 172.21.15.27:0/848772725 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4140f8fb0 con 0x7fa3f80692a0 2024-04-16T05:42:04.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.629+0000 7fa40be00700 1 -- 172.21.15.27:0/848772725 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7fa4140f8fb0 con 0x7fa3f80692a0 2024-04-16T05:42:04.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.631+0000 7fa41b000700 1 -- 172.21.15.27:0/848772725 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa3f80692a0 msgr2=0x7fa3f806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:04.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.631+0000 7fa41b000700 1 --2- 172.21.15.27:0/848772725 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa3f80692a0 0x7fa3f806b750 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa40000b970 tx=0x7fa400009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:04.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.631+0000 7fa41b000700 1 -- 172.21.15.27:0/848772725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4140662f0 msgr2=0x7fa4140fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:04.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.631+0000 7fa41b000700 1 --2- 172.21.15.27:0/848772725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4140662f0 0x7fa4140fd460 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa40400c890 tx=0x7fa4040044c0 comp rx=0 tx=0).stop 2024-04-16T05:42:04.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.632+0000 7fa41b000700 1 -- 172.21.15.27:0/848772725 shutdown_connections 2024-04-16T05:42:04.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.632+0000 7fa41b000700 1 --2- 172.21.15.27:0/848772725 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fa3f80692a0 0x7fa3f806b750 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:04.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.632+0000 7fa41b000700 1 --2- 172.21.15.27:0/848772725 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4140662f0 0x7fa4140fd460 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:04.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.632+0000 7fa41b000700 1 --2- 172.21.15.27:0/848772725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4140fff70 0x7fa4140fb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:04.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.632+0000 7fa41b000700 1 -- 172.21.15.27:0/848772725 >> 172.21.15.27:0/848772725 conn(0x7fa4140f74c0 msgr2=0x7fa4140f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:04.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.632+0000 7fa41b000700 1 -- 172.21.15.27:0/848772725 shutdown_connections 2024-04-16T05:42:04.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.632+0000 7fa41b000700 1 -- 172.21.15.27:0/848772725 wait complete. 2024-04-16T05:42:04.646 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:42:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.797+0000 7f23d2a00700 1 -- 172.21.15.27:0/2854937327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23cc0708f0 msgr2=0x7f23cc070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.797+0000 7f23d2a00700 1 --2- 172.21.15.27:0/2854937327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23cc0708f0 0x7f23cc070d60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f23bc004940 tx=0x7f23bc009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.797+0000 7f23d2a00700 1 -- 172.21.15.27:0/2854937327 shutdown_connections 2024-04-16T05:42:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.797+0000 7f23d2a00700 1 --2- 172.21.15.27:0/2854937327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23cc0708f0 0x7f23cc070d60 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.797+0000 7f23d2a00700 1 --2- 172.21.15.27:0/2854937327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23cc071ea0 0x7f23cc0703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.797+0000 7f23d2a00700 1 -- 172.21.15.27:0/2854937327 >> 172.21.15.27:0/2854937327 conn(0x7f23cc0f83a0 msgr2=0x7f23cc0fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.797+0000 7f23d2a00700 1 -- 172.21.15.27:0/2854937327 shutdown_connections 2024-04-16T05:42:04.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.797+0000 7f23d2a00700 1 -- 172.21.15.27:0/2854937327 wait complete. 2024-04-16T05:42:04.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.798+0000 7f23d2a00700 1 Processor -- start 2024-04-16T05:42:04.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.798+0000 7f23d2a00700 1 -- start start 2024-04-16T05:42:04.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.798+0000 7f23d2a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23cc0708f0 0x7f23cc113e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:04.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.798+0000 7f23d2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23cc071ea0 0x7f23cc1143b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:04.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.798+0000 7f23d2a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc1150e0 con 0x7f23cc071ea0 2024-04-16T05:42:04.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.798+0000 7f23d2a00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc115250 con 0x7f23cc0708f0 2024-04-16T05:42:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.799+0000 7f23d0c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23cc071ea0 0x7f23cc1143b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.799+0000 7f23d0c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23cc071ea0 0x7f23cc1143b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45806/0 (socket says 172.21.15.27:45806) 2024-04-16T05:42:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.799+0000 7f23d0c00700 1 -- 172.21.15.27:0/4211864255 learned_addr learned my addr 172.21.15.27:0/4211864255 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:42:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.799+0000 7f23d1600700 1 --2- 172.21.15.27:0/4211864255 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23cc0708f0 0x7f23cc113e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.799+0000 7f23d0c00700 1 -- 172.21.15.27:0/4211864255 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23cc0708f0 msgr2=0x7f23cc113e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.799+0000 7f23d0c00700 1 --2- 172.21.15.27:0/4211864255 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23cc0708f0 0x7f23cc113e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:04.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.799+0000 7f23d0c00700 1 -- 172.21.15.27:0/4211864255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23bc005040 con 0x7f23cc071ea0 2024-04-16T05:42:04.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.799+0000 7f23d0c00700 1 --2- 172.21.15.27:0/4211864255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23cc071ea0 0x7f23cc1143b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009150 tx=0x7f23bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:04.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.800+0000 7f23c9600700 1 -- 172.21.15.27:0/4211864255 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23bc00e040 con 0x7f23cc071ea0 2024-04-16T05:42:04.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.800+0000 7f23c9600700 1 -- 172.21.15.27:0/4211864255 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f23bc00bbf0 con 0x7f23cc071ea0 2024-04-16T05:42:04.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.800+0000 7f23c9600700 1 -- 172.21.15.27:0/4211864255 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23bc01c630 con 0x7f23cc071ea0 2024-04-16T05:42:04.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.800+0000 7f23d2a00700 1 -- 172.21.15.27:0/4211864255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23cc1154d0 con 0x7f23cc071ea0 2024-04-16T05:42:04.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.800+0000 7f23d2a00700 1 -- 172.21.15.27:0/4211864255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23cc115a90 con 0x7f23cc071ea0 2024-04-16T05:42:04.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.801+0000 7f23d2a00700 1 -- 172.21.15.27:0/4211864255 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23cc0fc020 con 0x7f23cc071ea0 2024-04-16T05:42:04.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.802+0000 7f23c9600700 1 -- 172.21.15.27:0/4211864255 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f23bc00bd60 con 0x7f23cc071ea0 2024-04-16T05:42:04.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.803+0000 7f23c9600700 1 --2- 172.21.15.27:0/4211864255 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f23b40692a0 0x7f23b406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:04.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.803+0000 7f23c9600700 1 -- 172.21.15.27:0/4211864255 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f23bc088610 con 0x7f23cc071ea0 2024-04-16T05:42:04.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.804+0000 7f23d1600700 1 --2- 172.21.15.27:0/4211864255 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f23b40692a0 0x7f23b406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:04.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.804+0000 7f23d1600700 1 --2- 172.21.15.27:0/4211864255 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f23b40692a0 0x7f23b406b750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f23cc0fc6a0 tx=0x7f23c000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:04.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:04.809+0000 7f23c9600700 1 -- 172.21.15.27:0/4211864255 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f23bc054980 con 0x7f23cc071ea0 2024-04-16T05:42:05.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:04 smithi122 ceph-mon[35998]: pgmap v281: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:05.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:04 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:42:05.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.050+0000 7f23d2a00700 1 -- 172.21.15.27:0/4211864255 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f23cc10bd20 con 0x7f23b40692a0 2024-04-16T05:42:05.049 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:04 smithi027 ceph-mon[29206]: pgmap v281: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:05.049 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:04 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:42:05.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.054+0000 7f23c9600700 1 -- 172.21.15.27:0/4211864255 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7f23cc10bd20 con 0x7f23b40692a0 2024-04-16T05:42:05.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.056+0000 7f23d2a00700 1 -- 172.21.15.27:0/4211864255 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f23b40692a0 msgr2=0x7f23b406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:05.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.056+0000 7f23d2a00700 1 --2- 172.21.15.27:0/4211864255 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f23b40692a0 0x7f23b406b750 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f23cc0fc6a0 tx=0x7f23c000b040 comp rx=0 tx=0).stop 2024-04-16T05:42:05.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.056+0000 7f23d2a00700 1 -- 172.21.15.27:0/4211864255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23cc071ea0 msgr2=0x7f23cc1143b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:05.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.056+0000 7f23d2a00700 1 --2- 172.21.15.27:0/4211864255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23cc071ea0 0x7f23cc1143b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009150 tx=0x7f23bc009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:05.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.057+0000 7f23d2a00700 1 -- 172.21.15.27:0/4211864255 shutdown_connections 2024-04-16T05:42:05.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.057+0000 7f23d2a00700 1 --2- 172.21.15.27:0/4211864255 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f23b40692a0 0x7f23b406b750 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:05.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.057+0000 7f23d2a00700 1 --2- 172.21.15.27:0/4211864255 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23cc0708f0 0x7f23cc113e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:05.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.057+0000 7f23d2a00700 1 --2- 172.21.15.27:0/4211864255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23cc071ea0 0x7f23cc1143b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:05.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.057+0000 7f23d2a00700 1 -- 172.21.15.27:0/4211864255 >> 172.21.15.27:0/4211864255 conn(0x7f23cc0f83a0 msgr2=0x7f23cc0f9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:05.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.057+0000 7f23d2a00700 1 -- 172.21.15.27:0/4211864255 shutdown_connections 2024-04-16T05:42:05.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.057+0000 7f23d2a00700 1 -- 172.21.15.27:0/4211864255 wait complete. 2024-04-16T05:42:05.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.222+0000 7f3de0600700 1 -- 172.21.15.27:0/462728365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dd8071ea0 msgr2=0x7f3dd80703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:05.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.222+0000 7f3de0600700 1 --2- 172.21.15.27:0/462728365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dd8071ea0 0x7f3dd80703b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc004940 tx=0x7f3dcc009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:05.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.223+0000 7f3de0600700 1 -- 172.21.15.27:0/462728365 shutdown_connections 2024-04-16T05:42:05.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.223+0000 7f3de0600700 1 --2- 172.21.15.27:0/462728365 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3dd80708f0 0x7f3dd8070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:05.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.223+0000 7f3de0600700 1 --2- 172.21.15.27:0/462728365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dd8071ea0 0x7f3dd80703b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:05.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.223+0000 7f3de0600700 1 -- 172.21.15.27:0/462728365 >> 172.21.15.27:0/462728365 conn(0x7f3dd80f8580 msgr2=0x7f3dd80fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:05.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.223+0000 7f3de0600700 1 -- 172.21.15.27:0/462728365 shutdown_connections 2024-04-16T05:42:05.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.223+0000 7f3de0600700 1 -- 172.21.15.27:0/462728365 wait complete. 2024-04-16T05:42:05.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.224+0000 7f3de0600700 1 Processor -- start 2024-04-16T05:42:05.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.224+0000 7f3de0600700 1 -- start start 2024-04-16T05:42:05.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.225+0000 7f3de0600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dd80708f0 0x7f3dd8113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:05.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.225+0000 7f3de0600700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3dd8071ea0 0x7f3dd81144f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:05.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.225+0000 7f3de0600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd8114af0 con 0x7f3dd80708f0 2024-04-16T05:42:05.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.225+0000 7f3de0600700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd8114c60 con 0x7f3dd8071ea0 2024-04-16T05:42:05.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.225+0000 7f3dddc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dd80708f0 0x7f3dd8113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:05.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.225+0000 7f3dddc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dd80708f0 0x7f3dd8113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45822/0 (socket says 172.21.15.27:45822) 2024-04-16T05:42:05.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.225+0000 7f3dddc00700 1 -- 172.21.15.27:0/1173770177 learned_addr learned my addr 172.21.15.27:0/1173770177 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:42:05.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.225+0000 7f3ddd200700 1 --2- 172.21.15.27:0/1173770177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3dd8071ea0 0x7f3dd81144f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:05.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.226+0000 7f3dddc00700 1 -- 172.21.15.27:0/1173770177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3dd8071ea0 msgr2=0x7f3dd81144f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:05.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.226+0000 7f3dddc00700 1 --2- 172.21.15.27:0/1173770177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3dd8071ea0 0x7f3dd81144f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:05.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.226+0000 7f3dddc00700 1 -- 172.21.15.27:0/1173770177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dcc005040 con 0x7f3dd80708f0 2024-04-16T05:42:05.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.226+0000 7f3ddd200700 1 --2- 172.21.15.27:0/1173770177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3dd8071ea0 0x7f3dd81144f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:42:05.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.226+0000 7f3dddc00700 1 --2- 172.21.15.27:0/1173770177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dd80708f0 0x7f3dd8113fb0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc00b970 tx=0x7f3dcc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:05.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.226+0000 7f3dd5600700 1 -- 172.21.15.27:0/1173770177 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3dcc00e040 con 0x7f3dd80708f0 2024-04-16T05:42:05.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.226+0000 7f3de0600700 1 -- 172.21.15.27:0/1173770177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dd8119660 con 0x7f3dd80708f0 2024-04-16T05:42:05.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.226+0000 7f3dd5600700 1 -- 172.21.15.27:0/1173770177 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3dcc004290 con 0x7f3dd80708f0 2024-04-16T05:42:05.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.226+0000 7f3dd5600700 1 -- 172.21.15.27:0/1173770177 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3dcc01c630 con 0x7f3dd80708f0 2024-04-16T05:42:05.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.227+0000 7f3de0600700 1 -- 172.21.15.27:0/1173770177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd8119c20 con 0x7f3dd80708f0 2024-04-16T05:42:05.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.228+0000 7f3de0600700 1 -- 172.21.15.27:0/1173770177 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dd804c710 con 0x7f3dd80708f0 2024-04-16T05:42:05.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.229+0000 7f3dd5600700 1 -- 172.21.15.27:0/1173770177 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f3dcc00bbf0 con 0x7f3dd80708f0 2024-04-16T05:42:05.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.230+0000 7f3dd5600700 1 --2- 172.21.15.27:0/1173770177 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f3dc00692a0 0x7f3dc006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:05.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.230+0000 7f3dd5600700 1 -- 172.21.15.27:0/1173770177 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f3dcc088570 con 0x7f3dd80708f0 2024-04-16T05:42:05.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.230+0000 7f3ddd200700 1 --2- 172.21.15.27:0/1173770177 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f3dc00692a0 0x7f3dc006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:05.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.231+0000 7f3ddd200700 1 --2- 172.21.15.27:0/1173770177 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f3dc00692a0 0x7f3dc006b750 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009fd0 tx=0x7f3dc400b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:05.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.235+0000 7f3dd5600700 1 -- 172.21.15.27:0/1173770177 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f3dcc0582c0 con 0x7f3dd80708f0 2024-04-16T05:42:05.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.517+0000 7f3de0600700 1 -- 172.21.15.27:0/1173770177 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3dd80ffaa0 con 0x7f3dc00692a0 2024-04-16T05:42:05.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.532+0000 7f3dd5600700 1 -- 172.21.15.27:0/1173770177 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7f3dd80ffaa0 con 0x7f3dc00692a0 2024-04-16T05:42:05.531 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:42:05.531 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (7m) 75s ago 9m 13.4M - 0.25.0 c8568f914cd2 95a43269c2c6 2024-04-16T05:42:05.531 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (9m) 75s ago 9m 6665k - 17.2.7-847-g45317f95 b333581ccdf5 f1acb6959c63 2024-04-16T05:42:05.531 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (7m) 80s ago 7m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 e81efc20e8c0 2024-04-16T05:42:05.531 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (6m) 75s ago 8m 73.3M - 9.4.7 954c08fa6188 0de7ee73579d 2024-04-16T05:42:05.531 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (100s) 75s ago 99s 12.8M - 17.2.7-847-g45317f95 b333581ccdf5 0824e2621be5 2024-04-16T05:42:05.531 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (109s) 75s ago 108s 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 b332a72cc594 2024-04-16T05:42:05.531 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (105s) 80s ago 105s 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 f5e36815ddfa 2024-04-16T05:42:05.531 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (96s) 80s ago 96s 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 755a73c65a7f 2024-04-16T05:42:05.531 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:9283 running (10m) 75s ago 10m 481M - 17.2.7-847-g45317f95 b333581ccdf5 78f461c198fe 2024-04-16T05:42:05.531 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283 running (7m) 80s ago 7m 434M - 17.2.7-847-g45317f95 b333581ccdf5 1ed822ae132b 2024-04-16T05:42:05.532 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (10m) 75s ago 11m 44.8M 2048M 17.2.7-847-g45317f95 b333581ccdf5 88575dbb8d98 2024-04-16T05:42:05.532 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (7m) 80s ago 7m 32.4M 2048M 17.2.7-847-g45317f95 b333581ccdf5 5df19eab6d21 2024-04-16T05:42:05.532 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (9m) 75s ago 9m 7763k - 1.5.0 0da6a335fe13 26bcc8d81c89 2024-04-16T05:42:05.532 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (7m) 80s ago 7m 7717k - 1.5.0 0da6a335fe13 7f796c66dff6 2024-04-16T05:42:05.532 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (6m) 75s ago 6m 44.8M 4288M 17.2.7-847-g45317f95 b333581ccdf5 15e049573bca 2024-04-16T05:42:05.532 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (6m) 75s ago 6m 45.1M 4288M 17.2.7-847-g45317f95 b333581ccdf5 e146295f6500 2024-04-16T05:42:05.532 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (5m) 75s ago 5m 46.0M 4288M 17.2.7-847-g45317f95 b333581ccdf5 6212f6a18fa4 2024-04-16T05:42:05.532 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (4m) 80s ago 4m 42.9M 5312M 17.2.7-847-g45317f95 b333581ccdf5 d6724ad05f9f 2024-04-16T05:42:05.532 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (4m) 80s ago 4m 41.1M 5312M 17.2.7-847-g45317f95 b333581ccdf5 7a59908a3e10 2024-04-16T05:42:05.532 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (3m) 80s ago 3m 40.5M 5312M 17.2.7-847-g45317f95 b333581ccdf5 8e4b60163907 2024-04-16T05:42:05.532 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (6m) 75s ago 8m 46.8M - 2.43.0 a07b618ecd1d 719577e1340d 2024-04-16T05:42:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.535+0000 7f3de0600700 1 -- 172.21.15.27:0/1173770177 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f3dc00692a0 msgr2=0x7f3dc006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.535+0000 7f3de0600700 1 --2- 172.21.15.27:0/1173770177 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f3dc00692a0 0x7f3dc006b750 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009fd0 tx=0x7f3dc400b040 comp rx=0 tx=0).stop 2024-04-16T05:42:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.535+0000 7f3de0600700 1 -- 172.21.15.27:0/1173770177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dd80708f0 msgr2=0x7f3dd8113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.535+0000 7f3de0600700 1 --2- 172.21.15.27:0/1173770177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dd80708f0 0x7f3dd8113fb0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc00b970 tx=0x7f3dcc009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.535+0000 7f3de0600700 1 -- 172.21.15.27:0/1173770177 shutdown_connections 2024-04-16T05:42:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.535+0000 7f3de0600700 1 --2- 172.21.15.27:0/1173770177 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f3dc00692a0 0x7f3dc006b750 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.535+0000 7f3de0600700 1 --2- 172.21.15.27:0/1173770177 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dd80708f0 0x7f3dd8113fb0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.535+0000 7f3de0600700 1 --2- 172.21.15.27:0/1173770177 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3dd8071ea0 0x7f3dd81144f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.535+0000 7f3de0600700 1 -- 172.21.15.27:0/1173770177 >> 172.21.15.27:0/1173770177 conn(0x7f3dd80f8580 msgr2=0x7f3dd80f9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.535+0000 7f3de0600700 1 -- 172.21.15.27:0/1173770177 shutdown_connections 2024-04-16T05:42:05.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.536+0000 7f3de0600700 1 -- 172.21.15.27:0/1173770177 wait complete. 2024-04-16T05:42:05.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.687+0000 7f0972200700 1 -- 172.21.15.27:0/1506786399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f096c1046d0 msgr2=0x7f096c104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:05.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.687+0000 7f0972200700 1 --2- 172.21.15.27:0/1506786399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f096c1046d0 0x7f096c104b40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0958004940 tx=0x7f0958009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:05.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.688+0000 7f0972200700 1 -- 172.21.15.27:0/1506786399 shutdown_connections 2024-04-16T05:42:05.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.688+0000 7f0972200700 1 --2- 172.21.15.27:0/1506786399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f096c1046d0 0x7f096c104b40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:05.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.688+0000 7f0972200700 1 --2- 172.21.15.27:0/1506786399 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f096c103420 0x7f096c103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:05.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.688+0000 7f0972200700 1 -- 172.21.15.27:0/1506786399 >> 172.21.15.27:0/1506786399 conn(0x7f096c072640 msgr2=0x7f096c074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:05.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.688+0000 7f0972200700 1 -- 172.21.15.27:0/1506786399 shutdown_connections 2024-04-16T05:42:05.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.688+0000 7f0972200700 1 -- 172.21.15.27:0/1506786399 wait complete. 2024-04-16T05:42:05.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.689+0000 7f0972200700 1 Processor -- start 2024-04-16T05:42:05.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.689+0000 7f0972200700 1 -- start start 2024-04-16T05:42:05.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.689+0000 7f0972200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f096c103420 0x7f096c113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:05.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.689+0000 7f0972200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f096c1046d0 0x7f096c114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:05.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.689+0000 7f0972200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096c114a90 con 0x7f096c103420 2024-04-16T05:42:05.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.689+0000 7f0972200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096c114c00 con 0x7f096c1046d0 2024-04-16T05:42:05.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.690+0000 7f096b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f096c103420 0x7f096c113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:05.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.690+0000 7f096b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f096c103420 0x7f096c113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45832/0 (socket says 172.21.15.27:45832) 2024-04-16T05:42:05.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.690+0000 7f096b400700 1 -- 172.21.15.27:0/4213055748 learned_addr learned my addr 172.21.15.27:0/4213055748 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:42:05.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.690+0000 7f096aa00700 1 --2- 172.21.15.27:0/4213055748 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f096c1046d0 0x7f096c114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:05.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.690+0000 7f096b400700 1 -- 172.21.15.27:0/4213055748 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f096c1046d0 msgr2=0x7f096c114490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:05.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.690+0000 7f096b400700 1 --2- 172.21.15.27:0/4213055748 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f096c1046d0 0x7f096c114490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:05.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.690+0000 7f096b400700 1 -- 172.21.15.27:0/4213055748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0958005040 con 0x7f096c103420 2024-04-16T05:42:05.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.690+0000 7f096b400700 1 --2- 172.21.15.27:0/4213055748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f096c103420 0x7f096c113f50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f095c002a50 tx=0x7f095c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:05.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.691+0000 7f0963400700 1 -- 172.21.15.27:0/4213055748 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f095c007cc0 con 0x7f096c103420 2024-04-16T05:42:05.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.691+0000 7f0972200700 1 -- 172.21.15.27:0/4213055748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f096c119660 con 0x7f096c103420 2024-04-16T05:42:05.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.691+0000 7f0963400700 1 -- 172.21.15.27:0/4213055748 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f095c00f040 con 0x7f096c103420 2024-04-16T05:42:05.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.691+0000 7f0963400700 1 -- 172.21.15.27:0/4213055748 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f095c009640 con 0x7f096c103420 2024-04-16T05:42:05.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.691+0000 7f0972200700 1 -- 172.21.15.27:0/4213055748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f096c119c20 con 0x7f096c103420 2024-04-16T05:42:05.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.692+0000 7f0972200700 1 -- 172.21.15.27:0/4213055748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f094c0052e0 con 0x7f096c103420 2024-04-16T05:42:05.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.694+0000 7f0963400700 1 -- 172.21.15.27:0/4213055748 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f095c0097a0 con 0x7f096c103420 2024-04-16T05:42:05.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.695+0000 7f0963400700 1 --2- 172.21.15.27:0/4213055748 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f09500692a0 0x7f095006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:05.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.695+0000 7f0963400700 1 -- 172.21.15.27:0/4213055748 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f095c086f80 con 0x7f096c103420 2024-04-16T05:42:05.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.695+0000 7f096aa00700 1 --2- 172.21.15.27:0/4213055748 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f09500692a0 0x7f095006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:05.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.696+0000 7f096aa00700 1 --2- 172.21.15.27:0/4213055748 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f09500692a0 0x7f095006b750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f095800b970 tx=0x7f0958009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:05.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:05.700+0000 7f0963400700 1 -- 172.21.15.27:0/4213055748 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f095c056c50 con 0x7f096c103420 2024-04-16T05:42:05.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:05 smithi027 ceph-mon[29206]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:42:06.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.012+0000 7f0972200700 1 -- 172.21.15.27:0/4213055748 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f094c005da0 con 0x7f096c103420 2024-04-16T05:42:06.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.013+0000 7f0963400700 1 -- 172.21.15.27:0/4213055748 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f095c0567e0 con 0x7f096c103420 2024-04-16T05:42:06.012 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:42:06.012 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:42:06.012 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T05:42:06.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:42:06.012 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:42:06.012 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T05:42:06.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:42:06.012 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:42:06.012 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T05:42:06.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:42:06.012 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:42:06.012 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T05:42:06.012 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:42:06.013 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:42:06.013 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 14 2024-04-16T05:42:06.013 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:42:06.013 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:42:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.017+0000 7f0972200700 1 -- 172.21.15.27:0/4213055748 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f09500692a0 msgr2=0x7f095006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.017+0000 7f0972200700 1 --2- 172.21.15.27:0/4213055748 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f09500692a0 0x7f095006b750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f095800b970 tx=0x7f0958009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.017+0000 7f0972200700 1 -- 172.21.15.27:0/4213055748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f096c103420 msgr2=0x7f096c113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.017+0000 7f0972200700 1 --2- 172.21.15.27:0/4213055748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f096c103420 0x7f096c113f50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f095c002a50 tx=0x7f095c0046b0 comp rx=0 tx=0).stop 2024-04-16T05:42:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.017+0000 7f0972200700 1 -- 172.21.15.27:0/4213055748 shutdown_connections 2024-04-16T05:42:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.017+0000 7f0972200700 1 --2- 172.21.15.27:0/4213055748 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f09500692a0 0x7f095006b750 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.017+0000 7f0972200700 1 --2- 172.21.15.27:0/4213055748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f096c103420 0x7f096c113f50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.017+0000 7f0972200700 1 --2- 172.21.15.27:0/4213055748 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f096c1046d0 0x7f096c114490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.017+0000 7f0972200700 1 -- 172.21.15.27:0/4213055748 >> 172.21.15.27:0/4213055748 conn(0x7f096c072640 msgr2=0x7f096c073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.017+0000 7f0972200700 1 -- 172.21.15.27:0/4213055748 shutdown_connections 2024-04-16T05:42:06.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.018+0000 7f0972200700 1 -- 172.21.15.27:0/4213055748 wait complete. 2024-04-16T05:42:06.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:05 smithi122 ceph-mon[35998]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:42:06.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.183+0000 7f0b6e200700 1 -- 172.21.15.27:0/1158931007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b680fff70 msgr2=0x7f0b68102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:06.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.183+0000 7f0b6e200700 1 --2- 172.21.15.27:0/1158931007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b680fff70 0x7f0b68102390 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0b54004940 tx=0x7f0b54009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:06.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.183+0000 7f0b6e200700 1 -- 172.21.15.27:0/1158931007 shutdown_connections 2024-04-16T05:42:06.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.183+0000 7f0b6e200700 1 --2- 172.21.15.27:0/1158931007 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b680fff70 0x7f0b68102390 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:06.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.183+0000 7f0b6e200700 1 --2- 172.21.15.27:0/1158931007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b680662f0 0x7f0b680ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:06.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.183+0000 7f0b6e200700 1 -- 172.21.15.27:0/1158931007 >> 172.21.15.27:0/1158931007 conn(0x7f0b680f74c0 msgr2=0x7f0b680f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:06.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.183+0000 7f0b6e200700 1 -- 172.21.15.27:0/1158931007 shutdown_connections 2024-04-16T05:42:06.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.183+0000 7f0b6e200700 1 -- 172.21.15.27:0/1158931007 wait complete. 2024-04-16T05:42:06.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.184+0000 7f0b6e200700 1 Processor -- start 2024-04-16T05:42:06.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.184+0000 7f0b6e200700 1 -- start start 2024-04-16T05:42:06.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.185+0000 7f0b6e200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b680662f0 0x7f0b6810fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:06.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.185+0000 7f0b6e200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b680fff70 0x7f0b68110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:06.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.185+0000 7f0b6e200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b68110850 con 0x7f0b680fff70 2024-04-16T05:42:06.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.185+0000 7f0b6e200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b68110f90 con 0x7f0b680662f0 2024-04-16T05:42:06.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.185+0000 7f0b66a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b680fff70 0x7f0b68110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:06.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.185+0000 7f0b66a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b680fff70 0x7f0b68110250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45854/0 (socket says 172.21.15.27:45854) 2024-04-16T05:42:06.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.185+0000 7f0b66a00700 1 -- 172.21.15.27:0/360826000 learned_addr learned my addr 172.21.15.27:0/360826000 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:42:06.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.185+0000 7f0b67400700 1 --2- 172.21.15.27:0/360826000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b680662f0 0x7f0b6810fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:06.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.186+0000 7f0b66a00700 1 -- 172.21.15.27:0/360826000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b680662f0 msgr2=0x7f0b6810fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:06.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.186+0000 7f0b66a00700 1 --2- 172.21.15.27:0/360826000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b680662f0 0x7f0b6810fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:06.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.186+0000 7f0b66a00700 1 -- 172.21.15.27:0/360826000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b54005040 con 0x7f0b680fff70 2024-04-16T05:42:06.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.186+0000 7f0b66a00700 1 --2- 172.21.15.27:0/360826000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b680fff70 0x7f0b68110250 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0b5400b3b0 tx=0x7f0b54009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:06.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.186+0000 7f0b5f400700 1 -- 172.21.15.27:0/360826000 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b5400e040 con 0x7f0b680fff70 2024-04-16T05:42:06.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.186+0000 7f0b5f400700 1 -- 172.21.15.27:0/360826000 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b54004290 con 0x7f0b680fff70 2024-04-16T05:42:06.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.186+0000 7f0b5f400700 1 -- 172.21.15.27:0/360826000 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b5401c630 con 0x7f0b680fff70 2024-04-16T05:42:06.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.187+0000 7f0b6e200700 1 -- 172.21.15.27:0/360826000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b68111210 con 0x7f0b680fff70 2024-04-16T05:42:06.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.187+0000 7f0b6e200700 1 -- 172.21.15.27:0/360826000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b681117d0 con 0x7f0b680fff70 2024-04-16T05:42:06.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.188+0000 7f0b6e200700 1 -- 172.21.15.27:0/360826000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b680f8ff0 con 0x7f0b680fff70 2024-04-16T05:42:06.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.189+0000 7f0b5f400700 1 -- 172.21.15.27:0/360826000 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f0b5400bbf0 con 0x7f0b680fff70 2024-04-16T05:42:06.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.190+0000 7f0b5f400700 1 --2- 172.21.15.27:0/360826000 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f0b4c0692b0 0x7f0b4c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:06.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.191+0000 7f0b5f400700 1 -- 172.21.15.27:0/360826000 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0b540885a0 con 0x7f0b680fff70 2024-04-16T05:42:06.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.191+0000 7f0b67400700 1 --2- 172.21.15.27:0/360826000 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f0b4c0692b0 0x7f0b4c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:06.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.192+0000 7f0b67400700 1 --2- 172.21.15.27:0/360826000 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f0b4c0692b0 0x7f0b4c06b760 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0b58007fd0 tx=0x7f0b58009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:06.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.199+0000 7f0b5f400700 1 -- 172.21.15.27:0/360826000 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f0b540583b0 con 0x7f0b680fff70 2024-04-16T05:42:06.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.473+0000 7f0b6e200700 1 -- 172.21.15.27:0/360826000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0b6805f5f0 con 0x7f0b680fff70 2024-04-16T05:42:06.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.475+0000 7f0b5f400700 1 -- 172.21.15.27:0/360826000 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1810 (secure 0 0 0) 0x7f0b54057f40 con 0x7f0b680fff70 2024-04-16T05:42:06.473 INFO:teuthology.orchestra.run.smithi027.stdout:e16 2024-04-16T05:42:06.474 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:42:06.474 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:42:06.474 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:42:06.474 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:42:06.474 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:42:06.474 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:42:06.474 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 15 2024-04-16T05:42:06.474 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:42:06.474 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:42:06.474 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:40:32.701649+0000 2024-04-16T05:42:06.474 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:42:06.474 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:42:06.474 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:42:06.474 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:42:06.474 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:42:06.475 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:42:06.475 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:42:06.475 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 46 2024-04-16T05:42:06.475 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:42:06.475 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:42:06.475 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:42:06.475 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=14462} 2024-04-16T05:42:06.475 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:42:06.475 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:42:06.475 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:42:06.475 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:42:06.475 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:42:06.475 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:42:06.475 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:42:06.475 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:42:06.475 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:42:06.476 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:42:06.476 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:42:06.476 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:42:06.476 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:42:06.476 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{-1:14474} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2065690722,v1:172.21.15.27:6827/2065690722] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:42:06.476 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:24263} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:42:06.476 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:24273} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:42:06.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.477+0000 7f0b6e200700 1 -- 172.21.15.27:0/360826000 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f0b4c0692b0 msgr2=0x7f0b4c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:06.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.477+0000 7f0b6e200700 1 --2- 172.21.15.27:0/360826000 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f0b4c0692b0 0x7f0b4c06b760 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0b58007fd0 tx=0x7f0b58009040 comp rx=0 tx=0).stop 2024-04-16T05:42:06.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.478+0000 7f0b6e200700 1 -- 172.21.15.27:0/360826000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b680fff70 msgr2=0x7f0b68110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:06.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.478+0000 7f0b6e200700 1 --2- 172.21.15.27:0/360826000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b680fff70 0x7f0b68110250 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0b5400b3b0 tx=0x7f0b54009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:06.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.478+0000 7f0b6e200700 1 -- 172.21.15.27:0/360826000 shutdown_connections 2024-04-16T05:42:06.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.478+0000 7f0b6e200700 1 --2- 172.21.15.27:0/360826000 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f0b4c0692b0 0x7f0b4c06b760 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:06.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.478+0000 7f0b6e200700 1 --2- 172.21.15.27:0/360826000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b680662f0 0x7f0b6810fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:06.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.478+0000 7f0b6e200700 1 --2- 172.21.15.27:0/360826000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b680fff70 0x7f0b68110250 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:06.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.478+0000 7f0b6e200700 1 -- 172.21.15.27:0/360826000 >> 172.21.15.27:0/360826000 conn(0x7f0b680f74c0 msgr2=0x7f0b680fbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:06.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.478+0000 7f0b6e200700 1 -- 172.21.15.27:0/360826000 shutdown_connections 2024-04-16T05:42:06.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.478+0000 7f0b6e200700 1 -- 172.21.15.27:0/360826000 wait complete. 2024-04-16T05:42:06.477 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2024-04-16T05:42:06.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.651+0000 7fd0e6800700 1 -- 172.21.15.27:0/3679436152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0e00fce80 msgr2=0x7fd0e00fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:06.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.651+0000 7fd0e6800700 1 --2- 172.21.15.27:0/3679436152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0e00fce80 0x7fd0e00fd290 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0009ce0 tx=0x7fd0d0004d10 comp rx=0 tx=0).stop 2024-04-16T05:42:06.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.651+0000 7fd0e6800700 1 -- 172.21.15.27:0/3679436152 shutdown_connections 2024-04-16T05:42:06.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.651+0000 7fd0e6800700 1 --2- 172.21.15.27:0/3679436152 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0e00fdff0 0x7fd0e00fe460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:06.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.651+0000 7fd0e6800700 1 --2- 172.21.15.27:0/3679436152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0e00fce80 0x7fd0e00fd290 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:06.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.651+0000 7fd0e6800700 1 -- 172.21.15.27:0/3679436152 >> 172.21.15.27:0/3679436152 conn(0x7fd0e00f8580 msgr2=0x7fd0e00fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.651+0000 7fd0e6800700 1 -- 172.21.15.27:0/3679436152 shutdown_connections 2024-04-16T05:42:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.651+0000 7fd0e6800700 1 -- 172.21.15.27:0/3679436152 wait complete. 2024-04-16T05:42:06.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.652+0000 7fd0e6800700 1 Processor -- start 2024-04-16T05:42:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.652+0000 7fd0e6800700 1 -- start start 2024-04-16T05:42:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.652+0000 7fd0e6800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0e00fce80 0x7fd0e010fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.652+0000 7fd0e6800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0e00fdff0 0x7fd0e0110200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.652+0000 7fd0e6800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e0110880 con 0x7fd0e00fce80 2024-04-16T05:42:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.652+0000 7fd0e6800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e0115220 con 0x7fd0e00fdff0 2024-04-16T05:42:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.653+0000 7fd0df400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0e00fce80 0x7fd0e010fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.653+0000 7fd0df400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0e00fce80 0x7fd0e010fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45874/0 (socket says 172.21.15.27:45874) 2024-04-16T05:42:06.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.653+0000 7fd0df400700 1 -- 172.21.15.27:0/1849407088 learned_addr learned my addr 172.21.15.27:0/1849407088 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:42:06.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.653+0000 7fd0d6a00700 1 --2- 172.21.15.27:0/1849407088 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0e00fdff0 0x7fd0e0110200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:06.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.653+0000 7fd0df400700 1 -- 172.21.15.27:0/1849407088 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0e00fdff0 msgr2=0x7fd0e0110200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:06.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.653+0000 7fd0df400700 1 --2- 172.21.15.27:0/1849407088 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0e00fdff0 0x7fd0e0110200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:06.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.653+0000 7fd0df400700 1 -- 172.21.15.27:0/1849407088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d0009940 con 0x7fd0e00fce80 2024-04-16T05:42:06.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.654+0000 7fd0df400700 1 --2- 172.21.15.27:0/1849407088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0e00fce80 0x7fd0e010fcc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0009380 tx=0x7fd0d0004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:06.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.654+0000 7fd0d7e00700 1 -- 172.21.15.27:0/1849407088 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0d000e040 con 0x7fd0e00fce80 2024-04-16T05:42:06.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.654+0000 7fd0d7e00700 1 -- 172.21.15.27:0/1849407088 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd0d00128e0 con 0x7fd0e00fce80 2024-04-16T05:42:06.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.654+0000 7fd0d7e00700 1 -- 172.21.15.27:0/1849407088 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0d001ca10 con 0x7fd0e00fce80 2024-04-16T05:42:06.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.654+0000 7fd0e6800700 1 -- 172.21.15.27:0/1849407088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e01153c0 con 0x7fd0e00fce80 2024-04-16T05:42:06.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.654+0000 7fd0e6800700 1 -- 172.21.15.27:0/1849407088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e0115900 con 0x7fd0e00fce80 2024-04-16T05:42:06.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.655+0000 7fd0e6800700 1 -- 172.21.15.27:0/1849407088 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e0101980 con 0x7fd0e00fce80 2024-04-16T05:42:06.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.656+0000 7fd0d7e00700 1 -- 172.21.15.27:0/1849407088 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fd0d0012a50 con 0x7fd0e00fce80 2024-04-16T05:42:06.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.657+0000 7fd0d7e00700 1 --2- 172.21.15.27:0/1849407088 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd0c40692a0 0x7fd0c406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.657+0000 7fd0d7e00700 1 -- 172.21.15.27:0/1849407088 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fd0d0088e60 con 0x7fd0e00fce80 2024-04-16T05:42:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.658+0000 7fd0d6a00700 1 --2- 172.21.15.27:0/1849407088 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd0c40692a0 0x7fd0c406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:06.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.658+0000 7fd0d6a00700 1 --2- 172.21.15.27:0/1849407088 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd0c40692a0 0x7fd0c406b750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0111190 tx=0x7fd0c800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:06.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.662+0000 7fd0d7e00700 1 -- 172.21.15.27:0/1849407088 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fd0d0055020 con 0x7fd0e00fce80 2024-04-16T05:42:06.900 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:06 smithi027 ceph-mon[29206]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:42:06.900 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:06 smithi027 ceph-mon[29206]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:42:06.900 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:06 smithi027 ceph-mon[29206]: pgmap v282: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:06.901 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:06 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/4213055748' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:42:06.901 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:06 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/360826000' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:42:06.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.901+0000 7fd0e6800700 1 -- 172.21.15.27:0/1849407088 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0e002b3b0 con 0x7fd0c40692a0 2024-04-16T05:42:06.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.904+0000 7fd0d7e00700 1 -- 172.21.15.27:0/1849407088 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+358 (secure 0 0 0) 0x7fd0e002b3b0 con 0x7fd0c40692a0 2024-04-16T05:42:06.902 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:42:06.903 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838", 2024-04-16T05:42:06.903 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:42:06.903 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:42:06.903 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [], 2024-04-16T05:42:06.903 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "", 2024-04-16T05:42:06.903 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:218a691417ef57c5d22a8d5cf3c5f41f7b542838 image", 2024-04-16T05:42:06.903 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:42:06.903 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:42:06.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.907+0000 7fd0e6800700 1 -- 172.21.15.27:0/1849407088 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd0c40692a0 msgr2=0x7fd0c406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:06.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.907+0000 7fd0e6800700 1 --2- 172.21.15.27:0/1849407088 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd0c40692a0 0x7fd0c406b750 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0111190 tx=0x7fd0c800b040 comp rx=0 tx=0).stop 2024-04-16T05:42:06.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.907+0000 7fd0e6800700 1 -- 172.21.15.27:0/1849407088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0e00fce80 msgr2=0x7fd0e010fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:06.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.907+0000 7fd0e6800700 1 --2- 172.21.15.27:0/1849407088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0e00fce80 0x7fd0e010fcc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0009380 tx=0x7fd0d0004d10 comp rx=0 tx=0).stop 2024-04-16T05:42:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.907+0000 7fd0e6800700 1 -- 172.21.15.27:0/1849407088 shutdown_connections 2024-04-16T05:42:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.907+0000 7fd0e6800700 1 --2- 172.21.15.27:0/1849407088 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fd0c40692a0 0x7fd0c406b750 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.907+0000 7fd0e6800700 1 --2- 172.21.15.27:0/1849407088 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd0e00fce80 0x7fd0e010fcc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.907+0000 7fd0e6800700 1 --2- 172.21.15.27:0/1849407088 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd0e00fdff0 0x7fd0e0110200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.907+0000 7fd0e6800700 1 -- 172.21.15.27:0/1849407088 >> 172.21.15.27:0/1849407088 conn(0x7fd0e00f8580 msgr2=0x7fd0e00f9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.908+0000 7fd0e6800700 1 -- 172.21.15.27:0/1849407088 shutdown_connections 2024-04-16T05:42:06.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:06.908+0000 7fd0e6800700 1 -- 172.21.15.27:0/1849407088 wait complete. 2024-04-16T05:42:07.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:06 smithi122 ceph-mon[35998]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:42:07.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:06 smithi122 ceph-mon[35998]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:42:07.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:06 smithi122 ceph-mon[35998]: pgmap v282: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:07.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:06 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/4213055748' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:42:07.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:06 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/360826000' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:42:07.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.069+0000 7faadde00700 1 -- 172.21.15.27:0/1436378791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad80662f0 msgr2=0x7faad80ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:07.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.069+0000 7faadde00700 1 --2- 172.21.15.27:0/1436378791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad80662f0 0x7faad80ffa30 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7faac8004940 tx=0x7faac8009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:07.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.069+0000 7faadde00700 1 -- 172.21.15.27:0/1436378791 shutdown_connections 2024-04-16T05:42:07.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.069+0000 7faadde00700 1 --2- 172.21.15.27:0/1436378791 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faad80fff70 0x7faad8102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:07.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.069+0000 7faadde00700 1 --2- 172.21.15.27:0/1436378791 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad80662f0 0x7faad80ffa30 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:07.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.069+0000 7faadde00700 1 -- 172.21.15.27:0/1436378791 >> 172.21.15.27:0/1436378791 conn(0x7faad80f74c0 msgr2=0x7faad80f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:07.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.069+0000 7faadde00700 1 -- 172.21.15.27:0/1436378791 shutdown_connections 2024-04-16T05:42:07.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.069+0000 7faadde00700 1 -- 172.21.15.27:0/1436378791 wait complete. 2024-04-16T05:42:07.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.070+0000 7faadde00700 1 Processor -- start 2024-04-16T05:42:07.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.070+0000 7faadde00700 1 -- start start 2024-04-16T05:42:07.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.071+0000 7faadde00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad80662f0 0x7faad810fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:07.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.071+0000 7faadde00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faad80fff70 0x7faad8110260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:07.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.071+0000 7faadde00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad8110860 con 0x7faad80662f0 2024-04-16T05:42:07.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.071+0000 7faadde00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad8110fa0 con 0x7faad80fff70 2024-04-16T05:42:07.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.071+0000 7faad6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad80662f0 0x7faad810fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:07.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.071+0000 7faad6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad80662f0 0x7faad810fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45880/0 (socket says 172.21.15.27:45880) 2024-04-16T05:42:07.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.071+0000 7faad6a00700 1 -- 172.21.15.27:0/4202179378 learned_addr learned my addr 172.21.15.27:0/4202179378 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:42:07.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.071+0000 7faad6a00700 1 -- 172.21.15.27:0/4202179378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faad80fff70 msgr2=0x7faad8110260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:07.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.071+0000 7faace000700 1 --2- 172.21.15.27:0/4202179378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faad80fff70 0x7faad8110260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:07.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.071+0000 7faad6a00700 1 --2- 172.21.15.27:0/4202179378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faad80fff70 0x7faad8110260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:07.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.071+0000 7faad6a00700 1 -- 172.21.15.27:0/4202179378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac8005040 con 0x7faad80662f0 2024-04-16T05:42:07.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.072+0000 7faad6a00700 1 --2- 172.21.15.27:0/4202179378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad80662f0 0x7faad810fd20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7faac800b970 tx=0x7faac8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:07.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.072+0000 7faacf400700 1 -- 172.21.15.27:0/4202179378 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faac800e040 con 0x7faad80662f0 2024-04-16T05:42:07.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.072+0000 7faacf400700 1 -- 172.21.15.27:0/4202179378 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faac8004290 con 0x7faad80662f0 2024-04-16T05:42:07.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.072+0000 7faacf400700 1 -- 172.21.15.27:0/4202179378 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faac801c630 con 0x7faad80662f0 2024-04-16T05:42:07.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.072+0000 7faadde00700 1 -- 172.21.15.27:0/4202179378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad8111220 con 0x7faad80662f0 2024-04-16T05:42:07.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.072+0000 7faadde00700 1 -- 172.21.15.27:0/4202179378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad81117e0 con 0x7faad80662f0 2024-04-16T05:42:07.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.073+0000 7faadde00700 1 -- 172.21.15.27:0/4202179378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad805f5f0 con 0x7faad80662f0 2024-04-16T05:42:07.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.074+0000 7faacf400700 1 -- 172.21.15.27:0/4202179378 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7faac800bbf0 con 0x7faad80662f0 2024-04-16T05:42:07.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.075+0000 7faacf400700 1 --2- 172.21.15.27:0/4202179378 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7faabc0692b0 0x7faabc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:07.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.076+0000 7faacf400700 1 -- 172.21.15.27:0/4202179378 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7faac80884b0 con 0x7faad80662f0 2024-04-16T05:42:07.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.076+0000 7faace000700 1 --2- 172.21.15.27:0/4202179378 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7faabc0692b0 0x7faabc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:07.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.077+0000 7faace000700 1 --2- 172.21.15.27:0/4202179378 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7faabc0692b0 0x7faabc06b760 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7faac0004d60 tx=0x7faac000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:07.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.082+0000 7faacf400700 1 -- 172.21.15.27:0/4202179378 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7faac80582c0 con 0x7faad80662f0 2024-04-16T05:42:07.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.388+0000 7faadde00700 1 -- 172.21.15.27:0/4202179378 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7faad81058e0 con 0x7faad80662f0 2024-04-16T05:42:07.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.389+0000 7faacf400700 1 -- 172.21.15.27:0/4202179378 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7faac8013e60 con 0x7faad80662f0 2024-04-16T05:42:07.387 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:42:07.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.392+0000 7faadde00700 1 -- 172.21.15.27:0/4202179378 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7faabc0692b0 msgr2=0x7faabc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:07.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.392+0000 7faadde00700 1 --2- 172.21.15.27:0/4202179378 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7faabc0692b0 0x7faabc06b760 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7faac0004d60 tx=0x7faac000b040 comp rx=0 tx=0).stop 2024-04-16T05:42:07.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.392+0000 7faadde00700 1 -- 172.21.15.27:0/4202179378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad80662f0 msgr2=0x7faad810fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:07.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.392+0000 7faadde00700 1 --2- 172.21.15.27:0/4202179378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad80662f0 0x7faad810fd20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7faac800b970 tx=0x7faac8009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:07.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.392+0000 7faadde00700 1 -- 172.21.15.27:0/4202179378 shutdown_connections 2024-04-16T05:42:07.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.392+0000 7faadde00700 1 --2- 172.21.15.27:0/4202179378 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7faabc0692b0 0x7faabc06b760 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:07.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.392+0000 7faadde00700 1 --2- 172.21.15.27:0/4202179378 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faad80662f0 0x7faad810fd20 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:07.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.392+0000 7faadde00700 1 --2- 172.21.15.27:0/4202179378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faad80fff70 0x7faad8110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:07.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.392+0000 7faadde00700 1 -- 172.21.15.27:0/4202179378 >> 172.21.15.27:0/4202179378 conn(0x7faad80f74c0 msgr2=0x7faad80fbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:07.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.392+0000 7faadde00700 1 -- 172.21.15.27:0/4202179378 shutdown_connections 2024-04-16T05:42:07.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:07.392+0000 7faadde00700 1 -- 172.21.15.27:0/4202179378 wait complete. 2024-04-16T05:42:08.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:07 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/4202179378' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:42:08.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:07 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:08.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:07 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:42:08.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:07 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:08.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:07 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/4202179378' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:42:08.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:07 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:08.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:07 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:42:08.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:07 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:08.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:08 smithi027 ceph-mon[29206]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:42:08.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:08 smithi027 ceph-mon[29206]: Upgrade: Target is version 19.0.0-2884-g218a6914 (unknown) 2024-04-16T05:42:08.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:08 smithi027 ceph-mon[29206]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa'] 2024-04-16T05:42:08.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:08 smithi027 ceph-mon[29206]: Upgrade: Need to upgrade myself (mgr.smithi027.qgzypi) 2024-04-16T05:42:08.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:08 smithi027 ceph-mon[29206]: pgmap v283: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:09.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:08 smithi122 ceph-mon[35998]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:42:09.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:08 smithi122 ceph-mon[35998]: Upgrade: Target is version 19.0.0-2884-g218a6914 (unknown) 2024-04-16T05:42:09.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:08 smithi122 ceph-mon[35998]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, digests ['quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa'] 2024-04-16T05:42:09.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:08 smithi122 ceph-mon[35998]: Upgrade: Need to upgrade myself (mgr.smithi027.qgzypi) 2024-04-16T05:42:09.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:08 smithi122 ceph-mon[35998]: pgmap v283: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:10.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:09 smithi122 ceph-mon[35998]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa on smithi122 2024-04-16T05:42:10.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:09 smithi027 ceph-mon[29206]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa on smithi122 2024-04-16T05:42:11.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:10 smithi122 ceph-mon[35998]: pgmap v284: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:11.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:10 smithi027 ceph-mon[29206]: pgmap v284: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:13.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:12 smithi122 ceph-mon[35998]: pgmap v285: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:13.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:12 smithi027 ceph-mon[29206]: pgmap v285: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:14 smithi122 ceph-mon[35998]: pgmap v286: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:15.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:14 smithi027 ceph-mon[29206]: pgmap v286: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:17.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:16 smithi122 ceph-mon[35998]: pgmap v287: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:17.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:16 smithi027 ceph-mon[29206]: pgmap v287: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:18.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:17 smithi122 ceph-mon[35998]: pgmap v288: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:18.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:17 smithi027 ceph-mon[29206]: pgmap v288: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:20.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:19 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:42:20.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:19 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:42:20.736 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:20 smithi122 ceph-mon[35998]: pgmap v289: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:21.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:20 smithi027 ceph-mon[29206]: pgmap v289: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:23.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:22 smithi122 ceph-mon[35998]: pgmap v290: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:23.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:22 smithi027 ceph-mon[29206]: pgmap v290: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:25.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:24 smithi122 ceph-mon[35998]: pgmap v291: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:25.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:24 smithi027 ceph-mon[29206]: pgmap v291: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:27.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:26 smithi122 ceph-mon[35998]: pgmap v292: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:27.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:26 smithi027 ceph-mon[29206]: pgmap v292: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:29.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:28 smithi122 ceph-mon[35998]: pgmap v293: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:29.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:28 smithi027 ceph-mon[29206]: pgmap v293: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:31.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:30 smithi122 ceph-mon[35998]: pgmap v294: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:31.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:30 smithi027 ceph-mon[29206]: pgmap v294: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:33.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:32 smithi122 ceph-mon[35998]: pgmap v295: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:33.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:32 smithi027 ceph-mon[29206]: pgmap v295: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:35.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:34 smithi122 ceph-mon[35998]: pgmap v296: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:35.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:34 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:42:35.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:34 smithi027 ceph-mon[29206]: pgmap v296: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:35.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:34 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:42:37.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:36 smithi122 ceph-mon[35998]: pgmap v297: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:37.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:36 smithi027 ceph-mon[29206]: pgmap v297: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:37.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.566+0000 7fb6af400700 1 -- 172.21.15.27:0/712644699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6a80fcef0 msgr2=0x7fb6a80fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:37.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.566+0000 7fb6af400700 1 --2- 172.21.15.27:0/712644699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6a80fcef0 0x7fb6a80fd320 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb69c004940 tx=0x7fb69c009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:37.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.567+0000 7fb6af400700 1 -- 172.21.15.27:0/712644699 shutdown_connections 2024-04-16T05:42:37.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.567+0000 7fb6af400700 1 --2- 172.21.15.27:0/712644699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6a80fe0e0 0x7fb6a80fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:37.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.567+0000 7fb6af400700 1 --2- 172.21.15.27:0/712644699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6a80fcef0 0x7fb6a80fd320 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:37.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.567+0000 7fb6af400700 1 -- 172.21.15.27:0/712644699 >> 172.21.15.27:0/712644699 conn(0x7fb6a80f8530 msgr2=0x7fb6a80fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:37.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.567+0000 7fb6af400700 1 -- 172.21.15.27:0/712644699 shutdown_connections 2024-04-16T05:42:37.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.567+0000 7fb6af400700 1 -- 172.21.15.27:0/712644699 wait complete. 2024-04-16T05:42:37.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.568+0000 7fb6af400700 1 Processor -- start 2024-04-16T05:42:37.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.568+0000 7fb6af400700 1 -- start start 2024-04-16T05:42:37.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.568+0000 7fb6af400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6a80fcef0 0x7fb6a810fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:37.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.568+0000 7fb6af400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6a80fe0e0 0x7fb6a8110270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:37.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.568+0000 7fb6af400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a8110870 con 0x7fb6a80fe0e0 2024-04-16T05:42:37.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.568+0000 7fb6af400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a8110fb0 con 0x7fb6a80fcef0 2024-04-16T05:42:37.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.568+0000 7fb6a7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6a80fe0e0 0x7fb6a8110270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:37.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.568+0000 7fb6a7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6a80fe0e0 0x7fb6a8110270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53970/0 (socket says 172.21.15.27:53970) 2024-04-16T05:42:37.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.569+0000 7fb6a7e00700 1 -- 172.21.15.27:0/1284258736 learned_addr learned my addr 172.21.15.27:0/1284258736 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:42:37.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.569+0000 7fb6aca00700 1 --2- 172.21.15.27:0/1284258736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6a80fcef0 0x7fb6a810fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:37.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.569+0000 7fb6a7e00700 1 -- 172.21.15.27:0/1284258736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6a80fcef0 msgr2=0x7fb6a810fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:37.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.569+0000 7fb6a7e00700 1 --2- 172.21.15.27:0/1284258736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6a80fcef0 0x7fb6a810fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:37.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.569+0000 7fb6a7e00700 1 -- 172.21.15.27:0/1284258736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69c005040 con 0x7fb6a80fe0e0 2024-04-16T05:42:37.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.569+0000 7fb6a7e00700 1 --2- 172.21.15.27:0/1284258736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6a80fe0e0 0x7fb6a8110270 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb694002a50 tx=0x7fb6940046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:37.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.570+0000 7fb6a4c00700 1 -- 172.21.15.27:0/1284258736 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb694007cc0 con 0x7fb6a80fe0e0 2024-04-16T05:42:37.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.570+0000 7fb6a4c00700 1 -- 172.21.15.27:0/1284258736 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb69400f040 con 0x7fb6a80fe0e0 2024-04-16T05:42:37.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.570+0000 7fb6a4c00700 1 -- 172.21.15.27:0/1284258736 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb69400da30 con 0x7fb6a80fe0e0 2024-04-16T05:42:37.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.570+0000 7fb6af400700 1 -- 172.21.15.27:0/1284258736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a8111290 con 0x7fb6a80fe0e0 2024-04-16T05:42:37.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.570+0000 7fb6af400700 1 -- 172.21.15.27:0/1284258736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a8111850 con 0x7fb6a80fe0e0 2024-04-16T05:42:37.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.571+0000 7fb6af400700 1 -- 172.21.15.27:0/1284258736 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a8101a70 con 0x7fb6a80fe0e0 2024-04-16T05:42:37.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.572+0000 7fb6a4c00700 1 -- 172.21.15.27:0/1284258736 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7fb69400db90 con 0x7fb6a80fe0e0 2024-04-16T05:42:37.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.573+0000 7fb6a4c00700 1 --2- 172.21.15.27:0/1284258736 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb6900692b0 0x7fb69006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:37.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.573+0000 7fb6a4c00700 1 -- 172.21.15.27:0/1284258736 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fb694086f60 con 0x7fb6a80fe0e0 2024-04-16T05:42:37.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.573+0000 7fb6aca00700 1 --2- 172.21.15.27:0/1284258736 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb6900692b0 0x7fb69006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:37.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.574+0000 7fb6aca00700 1 --2- 172.21.15.27:0/1284258736 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb6900692b0 0x7fb69006b760 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00b3b0 tx=0x7fb69c009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:37.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.578+0000 7fb6a4c00700 1 -- 172.21.15.27:0/1284258736 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7fb694056c30 con 0x7fb6a80fe0e0 2024-04-16T05:42:37.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.816+0000 7fb6af400700 1 -- 172.21.15.27:0/1284258736 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6a8111ac0 con 0x7fb6900692b0 2024-04-16T05:42:37.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.819+0000 7fb6a4c00700 1 -- 172.21.15.27:0/1284258736 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7fb6a8111ac0 con 0x7fb6900692b0 2024-04-16T05:42:37.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.822+0000 7fb6af400700 1 -- 172.21.15.27:0/1284258736 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb6900692b0 msgr2=0x7fb69006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:37.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.822+0000 7fb6af400700 1 --2- 172.21.15.27:0/1284258736 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb6900692b0 0x7fb69006b760 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00b3b0 tx=0x7fb69c009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:37.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.822+0000 7fb6af400700 1 -- 172.21.15.27:0/1284258736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6a80fe0e0 msgr2=0x7fb6a8110270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:37.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.822+0000 7fb6af400700 1 --2- 172.21.15.27:0/1284258736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6a80fe0e0 0x7fb6a8110270 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb694002a50 tx=0x7fb6940046b0 comp rx=0 tx=0).stop 2024-04-16T05:42:37.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.822+0000 7fb6af400700 1 -- 172.21.15.27:0/1284258736 shutdown_connections 2024-04-16T05:42:37.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.822+0000 7fb6af400700 1 --2- 172.21.15.27:0/1284258736 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7fb6900692b0 0x7fb69006b760 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:37.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.822+0000 7fb6af400700 1 --2- 172.21.15.27:0/1284258736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6a80fcef0 0x7fb6a810fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:37.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.822+0000 7fb6af400700 1 --2- 172.21.15.27:0/1284258736 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6a80fe0e0 0x7fb6a8110270 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:37.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.822+0000 7fb6af400700 1 -- 172.21.15.27:0/1284258736 >> 172.21.15.27:0/1284258736 conn(0x7fb6a80f8530 msgr2=0x7fb6a80f9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:37.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.823+0000 7fb6af400700 1 -- 172.21.15.27:0/1284258736 shutdown_connections 2024-04-16T05:42:37.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.823+0000 7fb6af400700 1 -- 172.21.15.27:0/1284258736 wait complete. 2024-04-16T05:42:37.837 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:42:37.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.981+0000 7f24afe00700 1 -- 172.21.15.27:0/3685560887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a81046b0 msgr2=0x7f24a8104b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:37.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.981+0000 7f24afe00700 1 --2- 172.21.15.27:0/3685560887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a81046b0 0x7f24a8104b20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2494004940 tx=0x7f2494009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:37.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.982+0000 7f24afe00700 1 -- 172.21.15.27:0/3685560887 shutdown_connections 2024-04-16T05:42:37.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.982+0000 7f24afe00700 1 --2- 172.21.15.27:0/3685560887 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a81046b0 0x7f24a8104b20 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:37.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.982+0000 7f24afe00700 1 --2- 172.21.15.27:0/3685560887 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24a8103400 0x7f24a8103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:37.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.982+0000 7f24afe00700 1 -- 172.21.15.27:0/3685560887 >> 172.21.15.27:0/3685560887 conn(0x7f24a80725e0 msgr2=0x7f24a8074a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:37.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.982+0000 7f24afe00700 1 -- 172.21.15.27:0/3685560887 shutdown_connections 2024-04-16T05:42:37.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.982+0000 7f24afe00700 1 -- 172.21.15.27:0/3685560887 wait complete. 2024-04-16T05:42:37.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.983+0000 7f24afe00700 1 Processor -- start 2024-04-16T05:42:37.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.983+0000 7f24afe00700 1 -- start start 2024-04-16T05:42:37.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.983+0000 7f24afe00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24a8103400 0x7f24a8113f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:37.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.983+0000 7f24afe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a81046b0 0x7f24a8114450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:37.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.983+0000 7f24afe00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a8114a50 con 0x7f24a81046b0 2024-04-16T05:42:37.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.983+0000 7f24afe00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a8114bc0 con 0x7f24a8103400 2024-04-16T05:42:37.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.984+0000 7f24aca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a81046b0 0x7f24a8114450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:37.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.984+0000 7f24aca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a81046b0 0x7f24a8114450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53984/0 (socket says 172.21.15.27:53984) 2024-04-16T05:42:37.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.984+0000 7f24aca00700 1 -- 172.21.15.27:0/1006367016 learned_addr learned my addr 172.21.15.27:0/1006367016 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:42:37.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.984+0000 7f24ad400700 1 --2- 172.21.15.27:0/1006367016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24a8103400 0x7f24a8113f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:37.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.984+0000 7f24aca00700 1 -- 172.21.15.27:0/1006367016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24a8103400 msgr2=0x7f24a8113f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:37.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.984+0000 7f24aca00700 1 --2- 172.21.15.27:0/1006367016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24a8103400 0x7f24a8113f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:37.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.984+0000 7f24aca00700 1 -- 172.21.15.27:0/1006367016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2494005040 con 0x7f24a81046b0 2024-04-16T05:42:37.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.984+0000 7f24aca00700 1 --2- 172.21.15.27:0/1006367016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a81046b0 0x7f24a8114450 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f249400b3b0 tx=0x7f2494009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:37.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.985+0000 7f24a5600700 1 -- 172.21.15.27:0/1006367016 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f249400e040 con 0x7f24a81046b0 2024-04-16T05:42:37.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.985+0000 7f24a5600700 1 -- 172.21.15.27:0/1006367016 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f249400bcc0 con 0x7f24a81046b0 2024-04-16T05:42:37.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.985+0000 7f24a5600700 1 -- 172.21.15.27:0/1006367016 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f249401c630 con 0x7f24a81046b0 2024-04-16T05:42:37.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.985+0000 7f24afe00700 1 -- 172.21.15.27:0/1006367016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a81195c0 con 0x7f24a81046b0 2024-04-16T05:42:37.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.985+0000 7f24afe00700 1 -- 172.21.15.27:0/1006367016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a8119b80 con 0x7f24a81046b0 2024-04-16T05:42:37.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.986+0000 7f24afe00700 1 -- 172.21.15.27:0/1006367016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a804c710 con 0x7f24a81046b0 2024-04-16T05:42:37.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.988+0000 7f24a5600700 1 -- 172.21.15.27:0/1006367016 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f2494013980 con 0x7f24a81046b0 2024-04-16T05:42:37.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.990+0000 7f24a5600700 1 --2- 172.21.15.27:0/1006367016 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f24900692a0 0x7f249006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:37.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.990+0000 7f24a5600700 1 -- 172.21.15.27:0/1006367016 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2494087fe0 con 0x7f24a81046b0 2024-04-16T05:42:37.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.990+0000 7f24ad400700 1 --2- 172.21.15.27:0/1006367016 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f24900692a0 0x7f249006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:37.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.991+0000 7f24ad400700 1 --2- 172.21.15.27:0/1006367016 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f24900692a0 0x7f249006b750 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f249c004990 tx=0x7f249c009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:37.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:37.996+0000 7f24a5600700 1 -- 172.21.15.27:0/1006367016 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f24940ba250 con 0x7f24a81046b0 2024-04-16T05:42:38.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.232+0000 7f24afe00700 1 -- 172.21.15.27:0/1006367016 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f24a8108990 con 0x7f24900692a0 2024-04-16T05:42:38.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.234+0000 7f24a5600700 1 -- 172.21.15.27:0/1006367016 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f24a8108990 con 0x7f24900692a0 2024-04-16T05:42:38.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.237+0000 7f24afe00700 1 -- 172.21.15.27:0/1006367016 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f24900692a0 msgr2=0x7f249006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:38.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.237+0000 7f24afe00700 1 --2- 172.21.15.27:0/1006367016 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f24900692a0 0x7f249006b750 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f249c004990 tx=0x7f249c009ee0 comp rx=0 tx=0).stop 2024-04-16T05:42:38.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.237+0000 7f24afe00700 1 -- 172.21.15.27:0/1006367016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a81046b0 msgr2=0x7f24a8114450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:38.236 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.237+0000 7f24afe00700 1 --2- 172.21.15.27:0/1006367016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a81046b0 0x7f24a8114450 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f249400b3b0 tx=0x7f2494009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:38.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.237+0000 7f24afe00700 1 -- 172.21.15.27:0/1006367016 shutdown_connections 2024-04-16T05:42:38.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.237+0000 7f24afe00700 1 --2- 172.21.15.27:0/1006367016 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f24900692a0 0x7f249006b750 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:38.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.237+0000 7f24afe00700 1 --2- 172.21.15.27:0/1006367016 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24a8103400 0x7f24a8113f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:38.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.237+0000 7f24afe00700 1 --2- 172.21.15.27:0/1006367016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24a81046b0 0x7f24a8114450 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:38.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.237+0000 7f24afe00700 1 -- 172.21.15.27:0/1006367016 >> 172.21.15.27:0/1006367016 conn(0x7f24a80725e0 msgr2=0x7f24a8073c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:38.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.238+0000 7f24afe00700 1 -- 172.21.15.27:0/1006367016 shutdown_connections 2024-04-16T05:42:38.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.238+0000 7f24afe00700 1 -- 172.21.15.27:0/1006367016 wait complete. 2024-04-16T05:42:38.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.399+0000 7f12c6800700 1 -- 172.21.15.27:0/1822171883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12c01046d0 msgr2=0x7f12c0104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:38.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.399+0000 7f12c6800700 1 --2- 172.21.15.27:0/1822171883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12c01046d0 0x7f12c0104b40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f12ac004940 tx=0x7f12ac009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:38.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.399+0000 7f12c6800700 1 -- 172.21.15.27:0/1822171883 shutdown_connections 2024-04-16T05:42:38.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.399+0000 7f12c6800700 1 --2- 172.21.15.27:0/1822171883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12c01046d0 0x7f12c0104b40 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:38.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.399+0000 7f12c6800700 1 --2- 172.21.15.27:0/1822171883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12c0103420 0x7f12c0103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:38.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.399+0000 7f12c6800700 1 -- 172.21.15.27:0/1822171883 >> 172.21.15.27:0/1822171883 conn(0x7f12c0072640 msgr2=0x7f12c0074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:38.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.399+0000 7f12c6800700 1 -- 172.21.15.27:0/1822171883 shutdown_connections 2024-04-16T05:42:38.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.399+0000 7f12c6800700 1 -- 172.21.15.27:0/1822171883 wait complete. 2024-04-16T05:42:38.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.400+0000 7f12c6800700 1 Processor -- start 2024-04-16T05:42:38.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.400+0000 7f12c6800700 1 -- start start 2024-04-16T05:42:38.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.401+0000 7f12c6800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12c0103420 0x7f12c0113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:38.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.401+0000 7f12c6800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12c01046d0 0x7f12c0114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:38.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.401+0000 7f12c6800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c0114a90 con 0x7f12c01046d0 2024-04-16T05:42:38.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.401+0000 7f12c6800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c0114c00 con 0x7f12c0103420 2024-04-16T05:42:38.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.401+0000 7f12bea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12c01046d0 0x7f12c0114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:38.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.401+0000 7f12bea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12c01046d0 0x7f12c0114490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54002/0 (socket says 172.21.15.27:54002) 2024-04-16T05:42:38.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.401+0000 7f12bea00700 1 -- 172.21.15.27:0/366083429 learned_addr learned my addr 172.21.15.27:0/366083429 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:42:38.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.401+0000 7f12bf400700 1 --2- 172.21.15.27:0/366083429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12c0103420 0x7f12c0113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:38.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.402+0000 7f12bea00700 1 -- 172.21.15.27:0/366083429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12c0103420 msgr2=0x7f12c0113f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:38.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.402+0000 7f12bea00700 1 --2- 172.21.15.27:0/366083429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12c0103420 0x7f12c0113f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:38.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.402+0000 7f12bea00700 1 -- 172.21.15.27:0/366083429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ac005040 con 0x7f12c01046d0 2024-04-16T05:42:38.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.402+0000 7f12bea00700 1 --2- 172.21.15.27:0/366083429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12c01046d0 0x7f12c0114490 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f12ac00b3b0 tx=0x7f12ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:38.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.402+0000 7f12b7400700 1 -- 172.21.15.27:0/366083429 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12ac00e040 con 0x7f12c01046d0 2024-04-16T05:42:38.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.402+0000 7f12b7400700 1 -- 172.21.15.27:0/366083429 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12ac004290 con 0x7f12c01046d0 2024-04-16T05:42:38.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.402+0000 7f12b7400700 1 -- 172.21.15.27:0/366083429 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12ac01c3f0 con 0x7f12c01046d0 2024-04-16T05:42:38.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.402+0000 7f12c6800700 1 -- 172.21.15.27:0/366083429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c0119600 con 0x7f12c01046d0 2024-04-16T05:42:38.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.403+0000 7f12c6800700 1 -- 172.21.15.27:0/366083429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c0119bc0 con 0x7f12c01046d0 2024-04-16T05:42:38.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.404+0000 7f12c6800700 1 -- 172.21.15.27:0/366083429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c004c710 con 0x7f12c01046d0 2024-04-16T05:42:38.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.405+0000 7f12b7400700 1 -- 172.21.15.27:0/366083429 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f12ac00bbf0 con 0x7f12c01046d0 2024-04-16T05:42:38.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.406+0000 7f12b7400700 1 --2- 172.21.15.27:0/366083429 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f12a40692a0 0x7f12a406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:38.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.406+0000 7f12b7400700 1 -- 172.21.15.27:0/366083429 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f12ac088510 con 0x7f12c01046d0 2024-04-16T05:42:38.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.406+0000 7f12bf400700 1 --2- 172.21.15.27:0/366083429 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f12a40692a0 0x7f12a406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:38.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.407+0000 7f12bf400700 1 --2- 172.21.15.27:0/366083429 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f12a40692a0 0x7f12a406b750 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f12b0007fd0 tx=0x7f12b0009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:38.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.410+0000 7f12b7400700 1 -- 172.21.15.27:0/366083429 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f12ac058260 con 0x7f12c01046d0 2024-04-16T05:42:38.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.625+0000 7f12c6800700 1 -- 172.21.15.27:0/366083429 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f12c0108a10 con 0x7f12a40692a0 2024-04-16T05:42:38.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.653+0000 7f12b7400700 1 -- 172.21.15.27:0/366083429 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f12c0108a10 con 0x7f12a40692a0 2024-04-16T05:42:38.652 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:42:38.653 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (7m) 108s ago 9m 13.4M - 0.25.0 c8568f914cd2 95a43269c2c6 2024-04-16T05:42:38.653 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (9m) 108s ago 9m 6665k - 17.2.7-847-g45317f95 b333581ccdf5 f1acb6959c63 2024-04-16T05:42:38.653 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (8m) 113s ago 8m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 e81efc20e8c0 2024-04-16T05:42:38.653 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (7m) 108s ago 9m 73.3M - 9.4.7 954c08fa6188 0de7ee73579d 2024-04-16T05:42:38.653 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (2m) 108s ago 2m 12.8M - 17.2.7-847-g45317f95 b333581ccdf5 0824e2621be5 2024-04-16T05:42:38.653 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (2m) 108s ago 2m 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 b332a72cc594 2024-04-16T05:42:38.653 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (2m) 113s ago 2m 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 f5e36815ddfa 2024-04-16T05:42:38.653 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (2m) 113s ago 2m 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 755a73c65a7f 2024-04-16T05:42:38.653 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:9283 running (11m) 108s ago 11m 481M - 17.2.7-847-g45317f95 b333581ccdf5 78f461c198fe 2024-04-16T05:42:38.653 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283 running (8m) 113s ago 8m 434M - 17.2.7-847-g45317f95 b333581ccdf5 1ed822ae132b 2024-04-16T05:42:38.653 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (11m) 108s ago 11m 44.8M 2048M 17.2.7-847-g45317f95 b333581ccdf5 88575dbb8d98 2024-04-16T05:42:38.653 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (8m) 113s ago 8m 32.4M 2048M 17.2.7-847-g45317f95 b333581ccdf5 5df19eab6d21 2024-04-16T05:42:38.653 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (9m) 108s ago 9m 7763k - 1.5.0 0da6a335fe13 26bcc8d81c89 2024-04-16T05:42:38.653 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (8m) 113s ago 8m 7717k - 1.5.0 0da6a335fe13 7f796c66dff6 2024-04-16T05:42:38.654 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (7m) 108s ago 7m 44.8M 4288M 17.2.7-847-g45317f95 b333581ccdf5 15e049573bca 2024-04-16T05:42:38.654 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (6m) 108s ago 6m 45.1M 4288M 17.2.7-847-g45317f95 b333581ccdf5 e146295f6500 2024-04-16T05:42:38.654 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (5m) 108s ago 5m 46.0M 4288M 17.2.7-847-g45317f95 b333581ccdf5 6212f6a18fa4 2024-04-16T05:42:38.654 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (5m) 113s ago 5m 42.9M 5312M 17.2.7-847-g45317f95 b333581ccdf5 d6724ad05f9f 2024-04-16T05:42:38.654 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (4m) 113s ago 4m 41.1M 5312M 17.2.7-847-g45317f95 b333581ccdf5 7a59908a3e10 2024-04-16T05:42:38.654 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (4m) 113s ago 4m 40.5M 5312M 17.2.7-847-g45317f95 b333581ccdf5 8e4b60163907 2024-04-16T05:42:38.654 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (7m) 108s ago 8m 46.8M - 2.43.0 a07b618ecd1d 719577e1340d 2024-04-16T05:42:38.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.655+0000 7f12c6800700 1 -- 172.21.15.27:0/366083429 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f12a40692a0 msgr2=0x7f12a406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:38.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.655+0000 7f12c6800700 1 --2- 172.21.15.27:0/366083429 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f12a40692a0 0x7f12a406b750 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f12b0007fd0 tx=0x7f12b0009040 comp rx=0 tx=0).stop 2024-04-16T05:42:38.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.655+0000 7f12c6800700 1 -- 172.21.15.27:0/366083429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12c01046d0 msgr2=0x7f12c0114490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:38.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.655+0000 7f12c6800700 1 --2- 172.21.15.27:0/366083429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12c01046d0 0x7f12c0114490 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f12ac00b3b0 tx=0x7f12ac009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:38.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.656+0000 7f12c6800700 1 -- 172.21.15.27:0/366083429 shutdown_connections 2024-04-16T05:42:38.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.656+0000 7f12c6800700 1 --2- 172.21.15.27:0/366083429 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f12a40692a0 0x7f12a406b750 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:38.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.656+0000 7f12c6800700 1 --2- 172.21.15.27:0/366083429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f12c0103420 0x7f12c0113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:38.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.656+0000 7f12c6800700 1 --2- 172.21.15.27:0/366083429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f12c01046d0 0x7f12c0114490 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:38.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.656+0000 7f12c6800700 1 -- 172.21.15.27:0/366083429 >> 172.21.15.27:0/366083429 conn(0x7f12c0072640 msgr2=0x7f12c0073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:38.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.656+0000 7f12c6800700 1 -- 172.21.15.27:0/366083429 shutdown_connections 2024-04-16T05:42:38.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.656+0000 7f12c6800700 1 -- 172.21.15.27:0/366083429 wait complete. 2024-04-16T05:42:38.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.810+0000 7f693d200700 1 -- 172.21.15.27:0/676179154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69380fdff0 msgr2=0x7f69380fe460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:38.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.810+0000 7f693d200700 1 --2- 172.21.15.27:0/676179154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69380fdff0 0x7f69380fe460 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6924004940 tx=0x7f6924009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:38.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.811+0000 7f693d200700 1 -- 172.21.15.27:0/676179154 shutdown_connections 2024-04-16T05:42:38.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.811+0000 7f693d200700 1 --2- 172.21.15.27:0/676179154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69380fdff0 0x7f69380fe460 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:38.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.811+0000 7f693d200700 1 --2- 172.21.15.27:0/676179154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69380fce80 0x7f69380fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:38.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.811+0000 7f693d200700 1 -- 172.21.15.27:0/676179154 >> 172.21.15.27:0/676179154 conn(0x7f69380f8580 msgr2=0x7f69380fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:38.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.811+0000 7f693d200700 1 -- 172.21.15.27:0/676179154 shutdown_connections 2024-04-16T05:42:38.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.811+0000 7f693d200700 1 -- 172.21.15.27:0/676179154 wait complete. 2024-04-16T05:42:38.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.812+0000 7f693d200700 1 Processor -- start 2024-04-16T05:42:38.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.812+0000 7f693d200700 1 -- start start 2024-04-16T05:42:38.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.812+0000 7f693d200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69380fce80 0x7f693810fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:38.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.812+0000 7f693d200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69380fdff0 0x7f6938110080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:38.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.812+0000 7f693d200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6938110680 con 0x7f69380fdff0 2024-04-16T05:42:38.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.812+0000 7f693d200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69381107f0 con 0x7f69380fce80 2024-04-16T05:42:38.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.812+0000 7f6935400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69380fdff0 0x7f6938110080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:38.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.812+0000 7f6935400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69380fdff0 0x7f6938110080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54024/0 (socket says 172.21.15.27:54024) 2024-04-16T05:42:38.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.812+0000 7f6935400700 1 -- 172.21.15.27:0/1544657267 learned_addr learned my addr 172.21.15.27:0/1544657267 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:42:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.813+0000 7f6935e00700 1 --2- 172.21.15.27:0/1544657267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69380fce80 0x7f693810fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.813+0000 7f6935400700 1 -- 172.21.15.27:0/1544657267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69380fce80 msgr2=0x7f693810fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.813+0000 7f6935400700 1 --2- 172.21.15.27:0/1544657267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69380fce80 0x7f693810fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.813+0000 7f6935400700 1 -- 172.21.15.27:0/1544657267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6924005040 con 0x7f69380fdff0 2024-04-16T05:42:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.813+0000 7f6935400700 1 --2- 172.21.15.27:0/1544657267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69380fdff0 0x7f6938110080 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f692400b3b0 tx=0x7f6924009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:38.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.813+0000 7f692e000700 1 -- 172.21.15.27:0/1544657267 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f692400e040 con 0x7f69380fdff0 2024-04-16T05:42:38.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.813+0000 7f693d200700 1 -- 172.21.15.27:0/1544657267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69381151f0 con 0x7f69380fdff0 2024-04-16T05:42:38.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.813+0000 7f692e000700 1 -- 172.21.15.27:0/1544657267 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6924004290 con 0x7f69380fdff0 2024-04-16T05:42:38.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.813+0000 7f692e000700 1 -- 172.21.15.27:0/1544657267 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f692401c3f0 con 0x7f69380fdff0 2024-04-16T05:42:38.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.814+0000 7f693d200700 1 -- 172.21.15.27:0/1544657267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69381157b0 con 0x7f69380fdff0 2024-04-16T05:42:38.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.815+0000 7f693d200700 1 -- 172.21.15.27:0/1544657267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6938101980 con 0x7f69380fdff0 2024-04-16T05:42:38.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.816+0000 7f692e000700 1 -- 172.21.15.27:0/1544657267 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f692400bbf0 con 0x7f69380fdff0 2024-04-16T05:42:38.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.816+0000 7f692e000700 1 --2- 172.21.15.27:0/1544657267 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f691c0692a0 0x7f691c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:38.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.817+0000 7f692e000700 1 -- 172.21.15.27:0/1544657267 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6924088300 con 0x7f69380fdff0 2024-04-16T05:42:38.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.817+0000 7f6935e00700 1 --2- 172.21.15.27:0/1544657267 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f691c0692a0 0x7f691c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:38.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.817+0000 7f6935e00700 1 --2- 172.21.15.27:0/1544657267 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f691c0692a0 0x7f691c06b750 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6928004f40 tx=0x7f692800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:38.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:38.821+0000 7f692e000700 1 -- 172.21.15.27:0/1544657267 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f6924058070 con 0x7f69380fdff0 2024-04-16T05:42:39.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:38 smithi122 ceph-mon[35998]: pgmap v298: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:39.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:38 smithi027 ceph-mon[29206]: pgmap v298: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:39.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.118+0000 7f693d200700 1 -- 172.21.15.27:0/1544657267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f693804c710 con 0x7f69380fdff0 2024-04-16T05:42:39.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.119+0000 7f692e000700 1 -- 172.21.15.27:0/1544657267 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f6924057c00 con 0x7f69380fdff0 2024-04-16T05:42:39.119 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:42:39.119 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:42:39.119 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T05:42:39.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:42:39.119 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:42:39.119 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T05:42:39.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:42:39.119 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:42:39.119 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T05:42:39.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:42:39.119 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:42:39.119 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T05:42:39.119 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:42:39.120 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:42:39.120 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 14 2024-04-16T05:42:39.120 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:42:39.120 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:42:39.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.122+0000 7f693d200700 1 -- 172.21.15.27:0/1544657267 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f691c0692a0 msgr2=0x7f691c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:39.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.122+0000 7f693d200700 1 --2- 172.21.15.27:0/1544657267 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f691c0692a0 0x7f691c06b750 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6928004f40 tx=0x7f692800b040 comp rx=0 tx=0).stop 2024-04-16T05:42:39.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.122+0000 7f693d200700 1 -- 172.21.15.27:0/1544657267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69380fdff0 msgr2=0x7f6938110080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:39.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.122+0000 7f693d200700 1 --2- 172.21.15.27:0/1544657267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69380fdff0 0x7f6938110080 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f692400b3b0 tx=0x7f6924009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:39.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.122+0000 7f693d200700 1 -- 172.21.15.27:0/1544657267 shutdown_connections 2024-04-16T05:42:39.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.122+0000 7f693d200700 1 --2- 172.21.15.27:0/1544657267 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f691c0692a0 0x7f691c06b750 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:39.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.122+0000 7f693d200700 1 --2- 172.21.15.27:0/1544657267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69380fce80 0x7f693810fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:39.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.122+0000 7f693d200700 1 --2- 172.21.15.27:0/1544657267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69380fdff0 0x7f6938110080 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:39.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.122+0000 7f693d200700 1 -- 172.21.15.27:0/1544657267 >> 172.21.15.27:0/1544657267 conn(0x7f69380f8580 msgr2=0x7f69380f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:39.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.122+0000 7f693d200700 1 -- 172.21.15.27:0/1544657267 shutdown_connections 2024-04-16T05:42:39.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.122+0000 7f693d200700 1 -- 172.21.15.27:0/1544657267 wait complete. 2024-04-16T05:42:39.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.279+0000 7f247e000700 1 -- 172.21.15.27:0/210860577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24780ff020 msgr2=0x7f24780ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:39.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.279+0000 7f247e000700 1 --2- 172.21.15.27:0/210860577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24780ff020 0x7f24780ff430 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2468004940 tx=0x7f2468009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:39.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.279+0000 7f247e000700 1 -- 172.21.15.27:0/210860577 shutdown_connections 2024-04-16T05:42:39.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.279+0000 7f247e000700 1 --2- 172.21.15.27:0/210860577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24781002d0 0x7f2478100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:39.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.279+0000 7f247e000700 1 --2- 172.21.15.27:0/210860577 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24780ff020 0x7f24780ff430 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:39.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.279+0000 7f247e000700 1 -- 172.21.15.27:0/210860577 >> 172.21.15.27:0/210860577 conn(0x7f24780fa950 msgr2=0x7f24780fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:39.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.279+0000 7f247e000700 1 -- 172.21.15.27:0/210860577 shutdown_connections 2024-04-16T05:42:39.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.279+0000 7f247e000700 1 -- 172.21.15.27:0/210860577 wait complete. 2024-04-16T05:42:39.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.280+0000 7f247e000700 1 Processor -- start 2024-04-16T05:42:39.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.280+0000 7f247e000700 1 -- start start 2024-04-16T05:42:39.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.281+0000 7f247e000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24780ff020 0x7f247810fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:39.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.281+0000 7f247e000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24781002d0 0x7f24781100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:39.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.281+0000 7f247e000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24781106a0 con 0x7f24780ff020 2024-04-16T05:42:39.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.281+0000 7f247e000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2478110810 con 0x7f24781002d0 2024-04-16T05:42:39.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.281+0000 7f2477400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24780ff020 0x7f247810fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:39.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.281+0000 7f2477400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24780ff020 0x7f247810fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54044/0 (socket says 172.21.15.27:54044) 2024-04-16T05:42:39.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.281+0000 7f2477400700 1 -- 172.21.15.27:0/1631558474 learned_addr learned my addr 172.21.15.27:0/1631558474 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:42:39.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.281+0000 7f2476a00700 1 --2- 172.21.15.27:0/1631558474 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24781002d0 0x7f24781100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:39.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.281+0000 7f2477400700 1 -- 172.21.15.27:0/1631558474 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24781002d0 msgr2=0x7f24781100a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:39.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.281+0000 7f2477400700 1 --2- 172.21.15.27:0/1631558474 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24781002d0 0x7f24781100a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:39.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.281+0000 7f2477400700 1 -- 172.21.15.27:0/1631558474 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2468005040 con 0x7f24780ff020 2024-04-16T05:42:39.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.282+0000 7f2477400700 1 --2- 172.21.15.27:0/1631558474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24780ff020 0x7f247810fb60 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f246800b3b0 tx=0x7f2468009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:39.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.282+0000 7f246f400700 1 -- 172.21.15.27:0/1631558474 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f246800e040 con 0x7f24780ff020 2024-04-16T05:42:39.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.282+0000 7f246f400700 1 -- 172.21.15.27:0/1631558474 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f246800bcc0 con 0x7f24780ff020 2024-04-16T05:42:39.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.282+0000 7f246f400700 1 -- 172.21.15.27:0/1631558474 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f246801c630 con 0x7f24780ff020 2024-04-16T05:42:39.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.282+0000 7f247e000700 1 -- 172.21.15.27:0/1631558474 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2478115210 con 0x7f24780ff020 2024-04-16T05:42:39.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.282+0000 7f247e000700 1 -- 172.21.15.27:0/1631558474 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2478115780 con 0x7f24780ff020 2024-04-16T05:42:39.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.283+0000 7f247e000700 1 -- 172.21.15.27:0/1631558474 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f247804c710 con 0x7f24780ff020 2024-04-16T05:42:39.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.284+0000 7f246f400700 1 -- 172.21.15.27:0/1631558474 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f2468013980 con 0x7f24780ff020 2024-04-16T05:42:39.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.285+0000 7f246f400700 1 --2- 172.21.15.27:0/1631558474 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f245c0692a0 0x7f245c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:39.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.285+0000 7f246f400700 1 -- 172.21.15.27:0/1631558474 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f24680880c0 con 0x7f24780ff020 2024-04-16T05:42:39.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.286+0000 7f2476a00700 1 --2- 172.21.15.27:0/1631558474 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f245c0692a0 0x7f245c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:39.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.286+0000 7f2476a00700 1 --2- 172.21.15.27:0/1631558474 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f245c0692a0 0x7f245c06b750 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2464004990 tx=0x7f2464009ee0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:39.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.291+0000 7f246f400700 1 -- 172.21.15.27:0/1631558474 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f2468057d10 con 0x7f24780ff020 2024-04-16T05:42:39.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.549+0000 7f247e000700 1 -- 172.21.15.27:0/1631558474 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2478115bf0 con 0x7f24780ff020 2024-04-16T05:42:39.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.550+0000 7f246f400700 1 -- 172.21.15.27:0/1631558474 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1810 (secure 0 0 0) 0x7f24680578a0 con 0x7f24780ff020 2024-04-16T05:42:39.549 INFO:teuthology.orchestra.run.smithi027.stdout:e16 2024-04-16T05:42:39.549 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:42:39.549 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:42:39.549 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:42:39.549 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:42:39.550 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:42:39.550 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:42:39.550 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 15 2024-04-16T05:42:39.550 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:42:39.550 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:42:39.550 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:40:32.701649+0000 2024-04-16T05:42:39.550 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:42:39.550 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:42:39.550 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:42:39.550 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:42:39.551 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:42:39.551 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:42:39.551 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:42:39.551 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 46 2024-04-16T05:42:39.551 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:42:39.551 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:42:39.551 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:42:39.551 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=14462} 2024-04-16T05:42:39.551 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:42:39.551 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:42:39.551 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:42:39.551 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:42:39.551 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:42:39.551 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:42:39.551 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:42:39.552 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:42:39.552 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:42:39.552 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:42:39.552 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:42:39.552 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:42:39.552 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:42:39.552 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{-1:14474} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2065690722,v1:172.21.15.27:6827/2065690722] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:42:39.552 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:24263} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:42:39.552 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:24273} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:42:39.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.553+0000 7f247e000700 1 -- 172.21.15.27:0/1631558474 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f245c0692a0 msgr2=0x7f245c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:39.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.553+0000 7f247e000700 1 --2- 172.21.15.27:0/1631558474 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f245c0692a0 0x7f245c06b750 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f2464004990 tx=0x7f2464009ee0 comp rx=0 tx=0).stop 2024-04-16T05:42:39.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.553+0000 7f247e000700 1 -- 172.21.15.27:0/1631558474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24780ff020 msgr2=0x7f247810fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:39.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.553+0000 7f247e000700 1 --2- 172.21.15.27:0/1631558474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24780ff020 0x7f247810fb60 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f246800b3b0 tx=0x7f2468009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:39.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.554+0000 7f247e000700 1 -- 172.21.15.27:0/1631558474 shutdown_connections 2024-04-16T05:42:39.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.554+0000 7f247e000700 1 --2- 172.21.15.27:0/1631558474 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f245c0692a0 0x7f245c06b750 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:39.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.554+0000 7f247e000700 1 --2- 172.21.15.27:0/1631558474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f24780ff020 0x7f247810fb60 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:39.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.554+0000 7f247e000700 1 --2- 172.21.15.27:0/1631558474 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f24781002d0 0x7f24781100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:39.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.554+0000 7f247e000700 1 -- 172.21.15.27:0/1631558474 >> 172.21.15.27:0/1631558474 conn(0x7f24780fa950 msgr2=0x7f24780fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:39.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.554+0000 7f247e000700 1 -- 172.21.15.27:0/1631558474 shutdown_connections 2024-04-16T05:42:39.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.554+0000 7f247e000700 1 -- 172.21.15.27:0/1631558474 wait complete. 2024-04-16T05:42:39.555 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2024-04-16T05:42:39.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.707+0000 7f0dddc00700 1 -- 172.21.15.27:0/329138662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dd8071ea0 msgr2=0x7f0dd80703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:39.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.708+0000 7f0dddc00700 1 --2- 172.21.15.27:0/329138662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dd8071ea0 0x7f0dd80703b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8004940 tx=0x7f0dc8009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:39.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.708+0000 7f0dddc00700 1 -- 172.21.15.27:0/329138662 shutdown_connections 2024-04-16T05:42:39.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.708+0000 7f0dddc00700 1 --2- 172.21.15.27:0/329138662 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dd80708f0 0x7f0dd8070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:39.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.708+0000 7f0dddc00700 1 --2- 172.21.15.27:0/329138662 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dd8071ea0 0x7f0dd80703b0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:39.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.708+0000 7f0dddc00700 1 -- 172.21.15.27:0/329138662 >> 172.21.15.27:0/329138662 conn(0x7f0dd80f83d0 msgr2=0x7f0dd80fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:39.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.708+0000 7f0dddc00700 1 -- 172.21.15.27:0/329138662 shutdown_connections 2024-04-16T05:42:39.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.708+0000 7f0dddc00700 1 -- 172.21.15.27:0/329138662 wait complete. 2024-04-16T05:42:39.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.709+0000 7f0dddc00700 1 Processor -- start 2024-04-16T05:42:39.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.709+0000 7f0dddc00700 1 -- start start 2024-04-16T05:42:39.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.710+0000 7f0dddc00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dd80708f0 0x7f0dd8113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:39.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.710+0000 7f0dddc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dd8071ea0 0x7f0dd8114460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:39.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.710+0000 7f0dddc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd81151d0 con 0x7f0dd8071ea0 2024-04-16T05:42:39.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.710+0000 7f0dddc00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd8115340 con 0x7f0dd80708f0 2024-04-16T05:42:39.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.710+0000 7f0dd7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dd8071ea0 0x7f0dd8114460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:39.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.710+0000 7f0dd7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dd8071ea0 0x7f0dd8114460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54068/0 (socket says 172.21.15.27:54068) 2024-04-16T05:42:39.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.710+0000 7f0dd7400700 1 -- 172.21.15.27:0/507110013 learned_addr learned my addr 172.21.15.27:0/507110013 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:42:39.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.710+0000 7f0dd7e00700 1 --2- 172.21.15.27:0/507110013 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dd80708f0 0x7f0dd8113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.710+0000 7f0dd7400700 1 -- 172.21.15.27:0/507110013 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dd80708f0 msgr2=0x7f0dd8113f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.710+0000 7f0dd7400700 1 --2- 172.21.15.27:0/507110013 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dd80708f0 0x7f0dd8113f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.710+0000 7f0dd7400700 1 -- 172.21.15.27:0/507110013 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc8005040 con 0x7f0dd8071ea0 2024-04-16T05:42:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.711+0000 7f0dd7400700 1 --2- 172.21.15.27:0/507110013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dd8071ea0 0x7f0dd8114460 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0dc400b790 tx=0x7f0dc4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.711+0000 7f0dcfe00700 1 -- 172.21.15.27:0/507110013 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dc400f770 con 0x7f0dd8071ea0 2024-04-16T05:42:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.711+0000 7f0dcfe00700 1 -- 172.21.15.27:0/507110013 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0dc400fda0 con 0x7f0dd8071ea0 2024-04-16T05:42:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.711+0000 7f0dcfe00700 1 -- 172.21.15.27:0/507110013 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0dc400d310 con 0x7f0dd8071ea0 2024-04-16T05:42:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.711+0000 7f0dddc00700 1 -- 172.21.15.27:0/507110013 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd81155c0 con 0x7f0dd8071ea0 2024-04-16T05:42:39.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.711+0000 7f0dddc00700 1 -- 172.21.15.27:0/507110013 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd8115b80 con 0x7f0dd8071ea0 2024-04-16T05:42:39.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.712+0000 7f0dddc00700 1 -- 172.21.15.27:0/507110013 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd805f5f0 con 0x7f0dd8071ea0 2024-04-16T05:42:39.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.713+0000 7f0dcfe00700 1 -- 172.21.15.27:0/507110013 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7f0dc4010420 con 0x7f0dd8071ea0 2024-04-16T05:42:39.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.714+0000 7f0dcfe00700 1 --2- 172.21.15.27:0/507110013 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f0dbc0692b0 0x7f0dbc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:39.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.714+0000 7f0dcfe00700 1 -- 172.21.15.27:0/507110013 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0dc40871c0 con 0x7f0dd8071ea0 2024-04-16T05:42:39.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.714+0000 7f0dd7e00700 1 --2- 172.21.15.27:0/507110013 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f0dbc0692b0 0x7f0dbc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:39.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.715+0000 7f0dd7e00700 1 --2- 172.21.15.27:0/507110013 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f0dbc0692b0 0x7f0dbc06b760 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8000c00 tx=0x7f0dc8009e30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:39.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.719+0000 7f0dcfe00700 1 -- 172.21.15.27:0/507110013 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7f0dc4053480 con 0x7f0dd8071ea0 2024-04-16T05:42:39.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:39 smithi027 ceph-mon[29206]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:42:39.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:39 smithi027 ceph-mon[29206]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:42:39.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:39 smithi027 ceph-mon[29206]: from='client.14612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:42:39.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:39 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1544657267' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:42:39.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:39 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1631558474' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:42:39.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.945+0000 7f0dddc00700 1 -- 172.21.15.27:0/507110013 --> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0dd8115ec0 con 0x7f0dbc0692b0 2024-04-16T05:42:39.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.949+0000 7f0dcfe00700 1 -- 172.21.15.27:0/507110013 <== mgr.14212 v2:172.21.15.27:6800/2203377971 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+447 (secure 0 0 0) 0x7f0dd8115ec0 con 0x7f0dbc0692b0 2024-04-16T05:42:39.947 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:42:39.947 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T05:42:39.947 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:42:39.947 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:42:39.947 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [], 2024-04-16T05:42:39.947 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "0/21 daemons upgraded", 2024-04-16T05:42:39.947 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa image on host smithi122", 2024-04-16T05:42:39.948 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:42:39.948 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:42:39.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.951+0000 7f0dddc00700 1 -- 172.21.15.27:0/507110013 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f0dbc0692b0 msgr2=0x7f0dbc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:39.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.951+0000 7f0dddc00700 1 --2- 172.21.15.27:0/507110013 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f0dbc0692b0 0x7f0dbc06b760 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0dc8000c00 tx=0x7f0dc8009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:39.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.951+0000 7f0dddc00700 1 -- 172.21.15.27:0/507110013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dd8071ea0 msgr2=0x7f0dd8114460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:39.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.951+0000 7f0dddc00700 1 --2- 172.21.15.27:0/507110013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dd8071ea0 0x7f0dd8114460 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0dc400b790 tx=0x7f0dc4009250 comp rx=0 tx=0).stop 2024-04-16T05:42:39.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.951+0000 7f0dddc00700 1 -- 172.21.15.27:0/507110013 shutdown_connections 2024-04-16T05:42:39.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.951+0000 7f0dddc00700 1 --2- 172.21.15.27:0/507110013 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7f0dbc0692b0 0x7f0dbc06b760 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:39.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.951+0000 7f0dddc00700 1 --2- 172.21.15.27:0/507110013 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0dd80708f0 0x7f0dd8113f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:39.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.951+0000 7f0dddc00700 1 --2- 172.21.15.27:0/507110013 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dd8071ea0 0x7f0dd8114460 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:39.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.951+0000 7f0dddc00700 1 -- 172.21.15.27:0/507110013 >> 172.21.15.27:0/507110013 conn(0x7f0dd80f83d0 msgr2=0x7f0dd80f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:39.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.951+0000 7f0dddc00700 1 -- 172.21.15.27:0/507110013 shutdown_connections 2024-04-16T05:42:39.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:39.951+0000 7f0dddc00700 1 -- 172.21.15.27:0/507110013 wait complete. 2024-04-16T05:42:40.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:39 smithi122 ceph-mon[35998]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:42:40.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:39 smithi122 ceph-mon[35998]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:42:40.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:39 smithi122 ceph-mon[35998]: from='client.14612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:42:40.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:39 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/1544657267' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:42:40.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:39 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/1631558474' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:42:40.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.110+0000 7ff7f6a00700 1 -- 172.21.15.27:0/3776274474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f00fcef0 msgr2=0x7ff7f00fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:40.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.110+0000 7ff7f6a00700 1 --2- 172.21.15.27:0/3776274474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f00fcef0 0x7ff7f00fd320 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0004940 tx=0x7ff7e0009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:40.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.110+0000 7ff7f6a00700 1 -- 172.21.15.27:0/3776274474 shutdown_connections 2024-04-16T05:42:40.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.110+0000 7ff7f6a00700 1 --2- 172.21.15.27:0/3776274474 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7f00fe0e0 0x7ff7f00fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:40.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.110+0000 7ff7f6a00700 1 --2- 172.21.15.27:0/3776274474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f00fcef0 0x7ff7f00fd320 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:40.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.110+0000 7ff7f6a00700 1 -- 172.21.15.27:0/3776274474 >> 172.21.15.27:0/3776274474 conn(0x7ff7f00f8530 msgr2=0x7ff7f00fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:40.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.110+0000 7ff7f6a00700 1 -- 172.21.15.27:0/3776274474 shutdown_connections 2024-04-16T05:42:40.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.110+0000 7ff7f6a00700 1 -- 172.21.15.27:0/3776274474 wait complete. 2024-04-16T05:42:40.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.111+0000 7ff7f6a00700 1 Processor -- start 2024-04-16T05:42:40.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.111+0000 7ff7f6a00700 1 -- start start 2024-04-16T05:42:40.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.112+0000 7ff7f6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f00fcef0 0x7ff7f0071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:40.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.112+0000 7ff7f6a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7f00fe0e0 0x7ff7f0070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:40.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.112+0000 7ff7f6a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f00708c0 con 0x7ff7f00fcef0 2024-04-16T05:42:40.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.112+0000 7ff7f6a00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f0070a30 con 0x7ff7f00fe0e0 2024-04-16T05:42:40.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.112+0000 7ff7efe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f00fcef0 0x7ff7f0071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:40.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.112+0000 7ff7efe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f00fcef0 0x7ff7f0071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54082/0 (socket says 172.21.15.27:54082) 2024-04-16T05:42:40.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.112+0000 7ff7efe00700 1 -- 172.21.15.27:0/1194829325 learned_addr learned my addr 172.21.15.27:0/1194829325 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:42:40.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.112+0000 7ff7ef400700 1 --2- 172.21.15.27:0/1194829325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7f00fe0e0 0x7ff7f0070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:40.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.112+0000 7ff7efe00700 1 -- 172.21.15.27:0/1194829325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7f00fe0e0 msgr2=0x7ff7f0070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:40.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.112+0000 7ff7efe00700 1 --2- 172.21.15.27:0/1194829325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7f00fe0e0 0x7ff7f0070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:40.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.112+0000 7ff7efe00700 1 -- 172.21.15.27:0/1194829325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e0005040 con 0x7ff7f00fcef0 2024-04-16T05:42:40.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.113+0000 7ff7efe00700 1 --2- 172.21.15.27:0/1194829325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f00fcef0 0x7ff7f0071e40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0000c00 tx=0x7ff7e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:40.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.113+0000 7ff7e7e00700 1 -- 172.21.15.27:0/1194829325 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7e000e040 con 0x7ff7f00fcef0 2024-04-16T05:42:40.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.113+0000 7ff7e7e00700 1 -- 172.21.15.27:0/1194829325 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff7e000bcc0 con 0x7ff7f00fcef0 2024-04-16T05:42:40.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.113+0000 7ff7f6a00700 1 -- 172.21.15.27:0/1194829325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7f0070cb0 con 0x7ff7f00fcef0 2024-04-16T05:42:40.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.113+0000 7ff7e7e00700 1 -- 172.21.15.27:0/1194829325 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7e001c630 con 0x7ff7f00fcef0 2024-04-16T05:42:40.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.113+0000 7ff7f6a00700 1 -- 172.21.15.27:0/1194829325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f0115a10 con 0x7ff7f00fcef0 2024-04-16T05:42:40.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.114+0000 7ff7f6a00700 1 -- 172.21.15.27:0/1194829325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7f005f5f0 con 0x7ff7f00fcef0 2024-04-16T05:42:40.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.115+0000 7ff7e7e00700 1 -- 172.21.15.27:0/1194829325 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 19) v1 ==== 86629+0+0 (secure 0 0 0) 0x7ff7e0013980 con 0x7ff7f00fcef0 2024-04-16T05:42:40.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.116+0000 7ff7e7e00700 1 --2- 172.21.15.27:0/1194829325 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7ff7d40692a0 0x7ff7d406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:42:40.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.116+0000 7ff7e7e00700 1 -- 172.21.15.27:0/1194829325 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff7e0088670 con 0x7ff7f00fcef0 2024-04-16T05:42:40.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.116+0000 7ff7ef400700 1 --2- 172.21.15.27:0/1194829325 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7ff7d40692a0 0x7ff7d406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:42:40.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.117+0000 7ff7ef400700 1 --2- 172.21.15.27:0/1194829325 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7ff7d40692a0 0x7ff7d406b750 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc004ab0 tx=0x7ff7dc007ab0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:42:40.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.121+0000 7ff7e7e00700 1 -- 172.21.15.27:0/1194829325 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176702 (secure 0 0 0) 0x7ff7e0054960 con 0x7ff7f00fcef0 2024-04-16T05:42:40.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.425+0000 7ff7f6a00700 1 -- 172.21.15.27:0/1194829325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff7f004bed0 con 0x7ff7f00fcef0 2024-04-16T05:42:40.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.426+0000 7ff7e7e00700 1 -- 172.21.15.27:0/1194829325 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7ff7e0057f80 con 0x7ff7f00fcef0 2024-04-16T05:42:40.424 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:42:40.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.429+0000 7ff7f6a00700 1 -- 172.21.15.27:0/1194829325 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7ff7d40692a0 msgr2=0x7ff7d406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:40.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.429+0000 7ff7f6a00700 1 --2- 172.21.15.27:0/1194829325 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7ff7d40692a0 0x7ff7d406b750 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc004ab0 tx=0x7ff7dc007ab0 comp rx=0 tx=0).stop 2024-04-16T05:42:40.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.429+0000 7ff7f6a00700 1 -- 172.21.15.27:0/1194829325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f00fcef0 msgr2=0x7ff7f0071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:42:40.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.429+0000 7ff7f6a00700 1 --2- 172.21.15.27:0/1194829325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f00fcef0 0x7ff7f0071e40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0000c00 tx=0x7ff7e0009e30 comp rx=0 tx=0).stop 2024-04-16T05:42:40.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.429+0000 7ff7f6a00700 1 -- 172.21.15.27:0/1194829325 shutdown_connections 2024-04-16T05:42:40.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.429+0000 7ff7f6a00700 1 --2- 172.21.15.27:0/1194829325 >> [v2:172.21.15.27:6800/2203377971,v1:172.21.15.27:6801/2203377971] conn(0x7ff7d40692a0 0x7ff7d406b750 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:40.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.430+0000 7ff7f6a00700 1 --2- 172.21.15.27:0/1194829325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f00fcef0 0x7ff7f0071e40 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:40.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.430+0000 7ff7f6a00700 1 --2- 172.21.15.27:0/1194829325 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7f00fe0e0 0x7ff7f0070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:42:40.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.430+0000 7ff7f6a00700 1 -- 172.21.15.27:0/1194829325 >> 172.21.15.27:0/1194829325 conn(0x7ff7f00f8530 msgr2=0x7ff7f00f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:42:40.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.430+0000 7ff7f6a00700 1 -- 172.21.15.27:0/1194829325 shutdown_connections 2024-04-16T05:42:40.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:42:40.430+0000 7ff7f6a00700 1 -- 172.21.15.27:0/1194829325 wait complete. 2024-04-16T05:42:41.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:40 smithi122 ceph-mon[35998]: pgmap v299: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:41.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:40 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/1194829325' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:42:41.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:40 smithi027 ceph-mon[29206]: pgmap v299: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:41.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:40 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1194829325' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:42:42.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:41 smithi122 ceph-mon[35998]: from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:42:42.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:41 smithi027 ceph-mon[29206]: from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:42:43.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:42 smithi122 ceph-mon[35998]: pgmap v300: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:43.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:42 smithi027 ceph-mon[29206]: pgmap v300: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:44.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:43 smithi122 ceph-mon[35998]: pgmap v301: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:44.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:43 smithi027 ceph-mon[29206]: pgmap v301: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:45.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:45 smithi122 ceph-mon[35998]: Upgrade: Updating mgr.smithi122.sgokkq 2024-04-16T05:42:45.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:45 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:45.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:45 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.sgokkq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T05:42:45.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:45 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T05:42:45.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:45 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:42:45.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:45 smithi122 ceph-mon[35998]: Deploying daemon mgr.smithi122.sgokkq on smithi122 2024-04-16T05:42:45.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:45 smithi027 ceph-mon[29206]: Upgrade: Updating mgr.smithi122.sgokkq 2024-04-16T05:42:45.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:45 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:45.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:45 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.sgokkq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T05:42:45.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:45 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T05:42:45.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:45 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:42:45.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:45 smithi027 ceph-mon[29206]: Deploying daemon mgr.smithi122.sgokkq on smithi122 2024-04-16T05:42:46.516 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:46 smithi122 ceph-mon[35998]: pgmap v302: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:46.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:46 smithi027 ceph-mon[29206]: pgmap v302: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:49.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:48 smithi122 ceph-mon[35998]: pgmap v303: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:49.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:48 smithi027 ceph-mon[29206]: pgmap v303: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:49.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:49 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:42:50.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:49 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:42:50.979 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:50 smithi122 ceph-mon[35998]: pgmap v304: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:50.980 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:50 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:50.980 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:50 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:50.980 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:50 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:42:50.980 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:50 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:50.980 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:50 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:51.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:50 smithi027 ceph-mon[29206]: pgmap v304: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:51.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:50 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:51.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:50 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:51.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:50 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:42:51.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:50 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:51.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:50 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:52.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:52 smithi122 ceph-mon[35998]: pgmap v305: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:53.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:52 smithi027 ceph-mon[29206]: pgmap v305: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:55.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:54 smithi122 ceph-mon[35998]: pgmap v306: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:55.079 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:54 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:55.079 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:54 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:55.080 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:54 smithi027 ceph-mon[29206]: pgmap v306: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:55.080 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:54 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:55.080 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:54 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:56.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:56 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:56.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:56 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:56.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:56 smithi122 ceph-mon[35998]: pgmap v307: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:56.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:56 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:56.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:56 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:42:56.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:56 smithi027 ceph-mon[29206]: pgmap v307: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:58.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:58 smithi122 ceph-mon[35998]: pgmap v308: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:58.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 12, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 62915 2024-04-16T05:42:58.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 12, success, outsize: 120 2024-04-16T05:42:58.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 14, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 62993 2024-04-16T05:42:58.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 14, success, outsize: 120 2024-04-16T05:42:59.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 16, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63074 2024-04-16T05:42:59.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 16, success, outsize: 120 2024-04-16T05:42:59.013 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 18, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63161 2024-04-16T05:42:59.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 18, success, outsize: 120 2024-04-16T05:42:59.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 20, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63245 2024-04-16T05:42:59.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 20, success, outsize: 120 2024-04-16T05:42:59.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 22, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63323 2024-04-16T05:42:59.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 22, success, outsize: 120 2024-04-16T05:42:59.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 24, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63404 2024-04-16T05:42:59.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 24, success, outsize: 120 2024-04-16T05:42:59.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:58 smithi027 ceph-mon[29206]: pgmap v308: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:42:59.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 26, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63491 2024-04-16T05:42:59.059 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 26, success, outsize: 120 2024-04-16T05:42:59.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 28, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63575 2024-04-16T05:42:59.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 28, success, outsize: 120 2024-04-16T05:42:59.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 30, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63653 2024-04-16T05:42:59.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 30, success, outsize: 120 2024-04-16T05:42:59.093 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 32, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63734 2024-04-16T05:42:59.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 32, success, outsize: 120 2024-04-16T05:42:59.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 34, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63821 2024-04-16T05:42:59.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 34, success, outsize: 120 2024-04-16T05:42:59.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 36, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63905 2024-04-16T05:42:59.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 36, success, outsize: 120 2024-04-16T05:42:59.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 38, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 63983 2024-04-16T05:42:59.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 38, success, outsize: 120 2024-04-16T05:42:59.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 40, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64064 2024-04-16T05:42:59.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 40, success, outsize: 120 2024-04-16T05:42:59.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 42, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64151 2024-04-16T05:42:59.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 42, success, outsize: 120 2024-04-16T05:42:59.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 44, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64235 2024-04-16T05:42:59.165 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 44, success, outsize: 120 2024-04-16T05:42:59.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 46, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64313 2024-04-16T05:42:59.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 46, success, outsize: 120 2024-04-16T05:42:59.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 48, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64394 2024-04-16T05:42:59.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 48, success, outsize: 120 2024-04-16T05:42:59.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 50, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64481 2024-04-16T05:42:59.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 50, success, outsize: 120 2024-04-16T05:42:59.217 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 52, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64565 2024-04-16T05:42:59.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 52, success, outsize: 120 2024-04-16T05:42:59.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 54, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64643 2024-04-16T05:42:59.231 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 54, success, outsize: 120 2024-04-16T05:42:59.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 56, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64724 2024-04-16T05:42:59.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 56, success, outsize: 120 2024-04-16T05:42:59.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 58, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64811 2024-04-16T05:42:59.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 58, success, outsize: 120 2024-04-16T05:42:59.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 60, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64895 2024-04-16T05:42:59.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 60, success, outsize: 120 2024-04-16T05:42:59.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 62, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 64973 2024-04-16T05:42:59.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 62, success, outsize: 120 2024-04-16T05:42:59.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 64, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 65054 2024-04-16T05:42:59.305 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 64, success, outsize: 120 2024-04-16T05:42:59.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 66, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 65141 2024-04-16T05:42:59.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 66, success, outsize: 120 2024-04-16T05:43:00.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:59 smithi122 ceph-mon[35998]: Standby manager daemon smithi122.sgokkq restarted 2024-04-16T05:43:00.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:59 smithi122 ceph-mon[35998]: Standby manager daemon smithi122.sgokkq started 2024-04-16T05:43:00.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:59 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.sgokkq/crt"}]: dispatch 2024-04-16T05:43:00.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:59 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T05:43:00.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:59 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.sgokkq/key"}]: dispatch 2024-04-16T05:43:00.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:42:59 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T05:43:00.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:59 smithi027 ceph-mon[29206]: Standby manager daemon smithi122.sgokkq restarted 2024-04-16T05:43:00.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:59 smithi027 ceph-mon[29206]: Standby manager daemon smithi122.sgokkq started 2024-04-16T05:43:00.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:59 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.sgokkq/crt"}]: dispatch 2024-04-16T05:43:00.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:59 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T05:43:00.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:59 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.sgokkq/key"}]: dispatch 2024-04-16T05:43:00.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:42:59 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T05:43:01.003 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:00 smithi122 ceph-mon[35998]: pgmap v309: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:01.004 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:00 smithi122 ceph-mon[35998]: mgrmap e20: smithi027.qgzypi(active, since 9m), standbys: smithi122.sgokkq 2024-04-16T05:43:01.004 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:00 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:43:01.004 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:00 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:43:01.004 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:00 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:43:01.004 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:00 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:43:01.004 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:00 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:43:01.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:00 smithi027 ceph-mon[29206]: pgmap v309: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:01.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:00 smithi027 ceph-mon[29206]: mgrmap e20: smithi027.qgzypi(active, since 9m), standbys: smithi122.sgokkq 2024-04-16T05:43:01.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:00 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:43:01.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:00 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:43:01.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:00 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:43:01.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:00 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:43:01.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:00 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' 2024-04-16T05:43:02.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:43:02.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr fail", "who": "smithi027.qgzypi"}]: dispatch 2024-04-16T05:43:02.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: Activating manager daemon smithi122.sgokkq 2024-04-16T05:43:02.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: osdmap e47: 6 total, 6 up, 6 in 2024-04-16T05:43:02.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "mgr fail", "who": "smithi027.qgzypi"}]': finished 2024-04-16T05:43:02.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: mgrmap e21: smithi122.sgokkq(active, starting, since 0.0164835s) 2024-04-16T05:43:02.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-16T05:43:02.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:43:02.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.ejoytk"}]: dispatch 2024-04-16T05:43:02.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.hkhpuq"}]: dispatch 2024-04-16T05:43:02.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.lojjdp"}]: dispatch 2024-04-16T05:43:02.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.rulewi"}]: dispatch 2024-04-16T05:43:02.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mgr metadata", "who": "smithi122.sgokkq", "id": "smithi122.sgokkq"}]: dispatch 2024-04-16T05:43:02.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T05:43:02.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T05:43:02.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T05:43:02.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T05:43:02.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T05:43:02.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T05:43:02.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T05:43:02.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T05:43:02.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T05:43:02.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: Manager daemon smithi122.sgokkq is now available 2024-04-16T05:43:02.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:43:02.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:43:02.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi122.sgokkq/mirror_snapshot_schedule"}]: dispatch 2024-04-16T05:43:02.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi122.sgokkq/trash_purge_schedule"}]: dispatch 2024-04-16T05:43:02.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:02.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:02.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:01 smithi122 ceph-mon[35998]: Deploying cephadm binary to smithi122 2024-04-16T05:43:02.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:43:02.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr fail", "who": "smithi027.qgzypi"}]: dispatch 2024-04-16T05:43:02.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: Activating manager daemon smithi122.sgokkq 2024-04-16T05:43:02.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: osdmap e47: 6 total, 6 up, 6 in 2024-04-16T05:43:02.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.14212 172.21.15.27:0/2230504008' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "mgr fail", "who": "smithi027.qgzypi"}]': finished 2024-04-16T05:43:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: mgrmap e21: smithi122.sgokkq(active, starting, since 0.0164835s) 2024-04-16T05:43:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-16T05:43:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:43:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.ejoytk"}]: dispatch 2024-04-16T05:43:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.hkhpuq"}]: dispatch 2024-04-16T05:43:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.lojjdp"}]: dispatch 2024-04-16T05:43:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.rulewi"}]: dispatch 2024-04-16T05:43:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mgr metadata", "who": "smithi122.sgokkq", "id": "smithi122.sgokkq"}]: dispatch 2024-04-16T05:43:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T05:43:02.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T05:43:02.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T05:43:02.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T05:43:02.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T05:43:02.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T05:43:02.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T05:43:02.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T05:43:02.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T05:43:02.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: Manager daemon smithi122.sgokkq is now available 2024-04-16T05:43:02.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:43:02.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:43:02.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi122.sgokkq/mirror_snapshot_schedule"}]: dispatch 2024-04-16T05:43:02.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi122.sgokkq/trash_purge_schedule"}]: dispatch 2024-04-16T05:43:02.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:02.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:02.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:01 smithi027 ceph-mon[29206]: Deploying cephadm binary to smithi122 2024-04-16T05:43:02.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:02 smithi027 ceph-mon[29206]: [16/Apr/2024:05:43:01] ENGINE Bus STARTING 2024-04-16T05:43:02.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:02 smithi027 ceph-mon[29206]: Deploying cephadm binary to smithi027 2024-04-16T05:43:02.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:02 smithi027 ceph-mon[29206]: [16/Apr/2024:05:43:01] ENGINE Serving on http://172.21.15.122:8765 2024-04-16T05:43:02.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:02 smithi027 ceph-mon[29206]: [16/Apr/2024:05:43:01] ENGINE Serving on https://172.21.15.122:7150 2024-04-16T05:43:02.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:02 smithi027 ceph-mon[29206]: [16/Apr/2024:05:43:01] ENGINE Bus STARTED 2024-04-16T05:43:02.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:02 smithi027 ceph-mon[29206]: mgrmap e22: smithi122.sgokkq(active, since 1.01687s) 2024-04-16T05:43:02.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:02 smithi027 ceph-mon[29206]: pgmap v3: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:03.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:02 smithi122 ceph-mon[35998]: [16/Apr/2024:05:43:01] ENGINE Bus STARTING 2024-04-16T05:43:03.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:02 smithi122 ceph-mon[35998]: Deploying cephadm binary to smithi027 2024-04-16T05:43:03.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:02 smithi122 ceph-mon[35998]: [16/Apr/2024:05:43:01] ENGINE Serving on http://172.21.15.122:8765 2024-04-16T05:43:03.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:02 smithi122 ceph-mon[35998]: [16/Apr/2024:05:43:01] ENGINE Serving on https://172.21.15.122:7150 2024-04-16T05:43:03.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:02 smithi122 ceph-mon[35998]: [16/Apr/2024:05:43:01] ENGINE Bus STARTED 2024-04-16T05:43:03.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:02 smithi122 ceph-mon[35998]: mgrmap e22: smithi122.sgokkq(active, since 1.01687s) 2024-04-16T05:43:03.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:02 smithi122 ceph-mon[35998]: pgmap v3: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:04.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:03 smithi027 ceph-mon[29206]: pgmap v4: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:04.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:03 smithi122 ceph-mon[35998]: pgmap v4: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:05.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:04 smithi027 ceph-mon[29206]: mgrmap e23: smithi122.sgokkq(active, since 3s) 2024-04-16T05:43:05.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:04 smithi122 ceph-mon[35998]: mgrmap e23: smithi122.sgokkq(active, since 3s) 2024-04-16T05:43:06.127 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:05 smithi027 ceph-mon[29206]: pgmap v5: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:06.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:05 smithi122 ceph-mon[35998]: pgmap v5: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:07.003 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:06 smithi027 ceph-mon[29206]: mgrmap e24: smithi122.sgokkq(active, since 5s) 2024-04-16T05:43:07.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:06 smithi122 ceph-mon[35998]: mgrmap e24: smithi122.sgokkq(active, since 5s) 2024-04-16T05:43:07.938 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:07 smithi027 ceph-mon[29206]: pgmap v6: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-04-16T05:43:08.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:07 smithi122 ceph-mon[35998]: pgmap v6: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-04-16T05:43:09.866 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:09 smithi027 ceph-mon[29206]: pgmap v7: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-16T05:43:10.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:09 smithi122 ceph-mon[35998]: pgmap v7: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-16T05:43:10.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.602+0000 7f18bce00700 1 -- 172.21.15.27:0/1277162252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18b8071ea0 msgr2=0x7f18b80703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:10.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.602+0000 7f18bce00700 1 --2- 172.21.15.27:0/1277162252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18b8071ea0 0x7f18b80703b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f18a8004940 tx=0x7f18a8009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:10.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.603+0000 7f18bce00700 1 -- 172.21.15.27:0/1277162252 shutdown_connections 2024-04-16T05:43:10.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.603+0000 7f18bce00700 1 --2- 172.21.15.27:0/1277162252 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18b80708f0 0x7f18b8070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:10.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.603+0000 7f18bce00700 1 --2- 172.21.15.27:0/1277162252 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18b8071ea0 0x7f18b80703b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:10.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.603+0000 7f18bce00700 1 -- 172.21.15.27:0/1277162252 >> 172.21.15.27:0/1277162252 conn(0x7f18b80f8530 msgr2=0x7f18b80fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:10.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.603+0000 7f18bce00700 1 -- 172.21.15.27:0/1277162252 shutdown_connections 2024-04-16T05:43:10.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.603+0000 7f18bce00700 1 -- 172.21.15.27:0/1277162252 wait complete. 2024-04-16T05:43:10.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.604+0000 7f18bce00700 1 Processor -- start 2024-04-16T05:43:10.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.604+0000 7f18bce00700 1 -- start start 2024-04-16T05:43:10.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.604+0000 7f18bce00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18b80708f0 0x7f18b810fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:10.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.604+0000 7f18bce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18b8071ea0 0x7f18b8110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:10.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.604+0000 7f18bce00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b8110f60 con 0x7f18b8071ea0 2024-04-16T05:43:10.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.604+0000 7f18bce00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b81110d0 con 0x7f18b80708f0 2024-04-16T05:43:10.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.605+0000 7f18b5400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18b8071ea0 0x7f18b8110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:10.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.605+0000 7f18b5400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18b8071ea0 0x7f18b8110250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57870/0 (socket says 172.21.15.27:57870) 2024-04-16T05:43:10.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.605+0000 7f18b5400700 1 -- 172.21.15.27:0/2972465882 learned_addr learned my addr 172.21.15.27:0/2972465882 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:43:10.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.605+0000 7f18b5e00700 1 --2- 172.21.15.27:0/2972465882 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18b80708f0 0x7f18b810fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:10.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.605+0000 7f18b5400700 1 -- 172.21.15.27:0/2972465882 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18b80708f0 msgr2=0x7f18b810fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:10.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.605+0000 7f18b5400700 1 --2- 172.21.15.27:0/2972465882 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18b80708f0 0x7f18b810fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:10.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.605+0000 7f18b5400700 1 -- 172.21.15.27:0/2972465882 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a8005040 con 0x7f18b8071ea0 2024-04-16T05:43:10.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.606+0000 7f18b5400700 1 --2- 172.21.15.27:0/2972465882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18b8071ea0 0x7f18b8110250 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f18a400b790 tx=0x7f18a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:10.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.606+0000 7f18ae000700 1 -- 172.21.15.27:0/2972465882 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18a400f770 con 0x7f18b8071ea0 2024-04-16T05:43:10.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.606+0000 7f18ae000700 1 -- 172.21.15.27:0/2972465882 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18a400fda0 con 0x7f18b8071ea0 2024-04-16T05:43:10.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.606+0000 7f18bce00700 1 -- 172.21.15.27:0/2972465882 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b81113b0 con 0x7f18b8071ea0 2024-04-16T05:43:10.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.606+0000 7f18ae000700 1 -- 172.21.15.27:0/2972465882 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18a400d310 con 0x7f18b8071ea0 2024-04-16T05:43:10.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.606+0000 7f18bce00700 1 -- 172.21.15.27:0/2972465882 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b8111970 con 0x7f18b8071ea0 2024-04-16T05:43:10.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.607+0000 7f18bce00700 1 -- 172.21.15.27:0/2972465882 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b805f5f0 con 0x7f18b8071ea0 2024-04-16T05:43:10.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.608+0000 7f18ae000700 1 -- 172.21.15.27:0/2972465882 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 24) v1 ==== 46568+0+0 (secure 0 0 0) 0x7f18a400f8d0 con 0x7f18b8071ea0 2024-04-16T05:43:10.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.608+0000 7f18ae000700 1 --2- 172.21.15.27:0/2972465882 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f189c039470 0x7f189c03b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:10.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.608+0000 7f18ae000700 1 -- 172.21.15.27:0/2972465882 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f18a404d120 con 0x7f18b8071ea0 2024-04-16T05:43:10.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.609+0000 7f18b5e00700 1 --2- 172.21.15.27:0/2972465882 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f189c039470 0x7f189c03b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:10.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.610+0000 7f18b5e00700 1 --2- 172.21.15.27:0/2972465882 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f189c039470 0x7f189c03b920 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f18a800b970 tx=0x7f18a8009e30 comp rx=0 tx=0).ready entity=mgr.24369 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:10.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.615+0000 7f18ae000700 1 -- 172.21.15.27:0/2972465882 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f18a400e4c0 con 0x7f18b8071ea0 2024-04-16T05:43:10.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.850+0000 7f18ae000700 1 -- 172.21.15.27:0/2972465882 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 25) v1 ==== 89652+0+0 (secure 0 0 0) 0x7f18a4017c10 con 0x7f18b8071ea0 2024-04-16T05:43:10.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.872+0000 7f18bce00700 1 -- 172.21.15.27:0/2972465882 --> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f18b8107be0 con 0x7f189c039470 2024-04-16T05:43:10.871 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:10 smithi027 ceph-mon[29206]: Standby manager daemon smithi027.qgzypi started 2024-04-16T05:43:10.871 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:10 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.27:0/201953492' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.qgzypi/crt"}]: dispatch 2024-04-16T05:43:10.872 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:10 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.27:0/201953492' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T05:43:10.872 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:10 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.27:0/201953492' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.qgzypi/key"}]: dispatch 2024-04-16T05:43:10.872 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:10 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.27:0/201953492' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T05:43:10.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.877+0000 7f18ae000700 1 -- 172.21.15.27:0/2972465882 <== mgr.24369 v2:172.21.15.122:6828/871482489 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7f18b8107be0 con 0x7f189c039470 2024-04-16T05:43:10.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.880+0000 7f18bce00700 1 -- 172.21.15.27:0/2972465882 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f189c039470 msgr2=0x7f189c03b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:10.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.880+0000 7f18bce00700 1 --2- 172.21.15.27:0/2972465882 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f189c039470 0x7f189c03b920 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f18a800b970 tx=0x7f18a8009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:10.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.880+0000 7f18bce00700 1 -- 172.21.15.27:0/2972465882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18b8071ea0 msgr2=0x7f18b8110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:10.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.880+0000 7f18bce00700 1 --2- 172.21.15.27:0/2972465882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18b8071ea0 0x7f18b8110250 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f18a400b790 tx=0x7f18a4009250 comp rx=0 tx=0).stop 2024-04-16T05:43:10.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.880+0000 7f18bce00700 1 -- 172.21.15.27:0/2972465882 shutdown_connections 2024-04-16T05:43:10.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.880+0000 7f18bce00700 1 --2- 172.21.15.27:0/2972465882 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f18b80708f0 0x7f18b810fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:10.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.880+0000 7f18bce00700 1 --2- 172.21.15.27:0/2972465882 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f189c039470 0x7f189c03b920 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:10.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.880+0000 7f18bce00700 1 --2- 172.21.15.27:0/2972465882 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18b8071ea0 0x7f18b8110250 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:10.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.880+0000 7f18bce00700 1 -- 172.21.15.27:0/2972465882 >> 172.21.15.27:0/2972465882 conn(0x7f18b80f8530 msgr2=0x7f18b80f9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:10.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.880+0000 7f18bce00700 1 -- 172.21.15.27:0/2972465882 shutdown_connections 2024-04-16T05:43:10.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:10.880+0000 7f18bce00700 1 -- 172.21.15.27:0/2972465882 wait complete. 2024-04-16T05:43:10.894 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:43:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.039+0000 7f3641800700 1 -- 172.21.15.27:0/2698487712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f363c0fe0e0 msgr2=0x7f363c0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.039+0000 7f3641800700 1 --2- 172.21.15.27:0/2698487712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f363c0fe0e0 0x7f363c0fe550 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3628004940 tx=0x7f3628009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.040+0000 7f3641800700 1 -- 172.21.15.27:0/2698487712 shutdown_connections 2024-04-16T05:43:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.040+0000 7f3641800700 1 --2- 172.21.15.27:0/2698487712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f363c0fe0e0 0x7f363c0fe550 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.040+0000 7f3641800700 1 --2- 172.21.15.27:0/2698487712 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f363c0fcef0 0x7f363c0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.040+0000 7f3641800700 1 -- 172.21.15.27:0/2698487712 >> 172.21.15.27:0/2698487712 conn(0x7f363c0f8530 msgr2=0x7f363c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.040+0000 7f3641800700 1 -- 172.21.15.27:0/2698487712 shutdown_connections 2024-04-16T05:43:11.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.040+0000 7f3641800700 1 -- 172.21.15.27:0/2698487712 wait complete. 2024-04-16T05:43:11.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.041+0000 7f3641800700 1 Processor -- start 2024-04-16T05:43:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.042+0000 7f3641800700 1 -- start start 2024-04-16T05:43:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.042+0000 7f3641800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f363c0fcef0 0x7f363c10fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.042+0000 7f3641800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f363c0fe0e0 0x7f363c1101a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.042+0000 7f3641800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363c1107a0 con 0x7f363c0fcef0 2024-04-16T05:43:11.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.042+0000 7f3641800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363c110ff0 con 0x7f363c0fe0e0 2024-04-16T05:43:11.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.043+0000 7f363a000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f363c0fe0e0 0x7f363c1101a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:11.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.043+0000 7f363a000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f363c0fe0e0 0x7f363c1101a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:44858/0 (socket says 172.21.15.27:44858) 2024-04-16T05:43:11.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.043+0000 7f363a000700 1 -- 172.21.15.27:0/1682670819 learned_addr learned my addr 172.21.15.27:0/1682670819 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:43:11.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.043+0000 7f363aa00700 1 --2- 172.21.15.27:0/1682670819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f363c0fcef0 0x7f363c10fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:11.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.044+0000 7f363a000700 1 -- 172.21.15.27:0/1682670819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f363c0fcef0 msgr2=0x7f363c10fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:11.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.044+0000 7f363a000700 1 --2- 172.21.15.27:0/1682670819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f363c0fcef0 0x7f363c10fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:11.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.044+0000 7f363a000700 1 -- 172.21.15.27:0/1682670819 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3628005040 con 0x7f363c0fe0e0 2024-04-16T05:43:11.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.044+0000 7f363a000700 1 --2- 172.21.15.27:0/1682670819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f363c0fe0e0 0x7f363c1101a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3628009150 tx=0x7f3628009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.045+0000 7f3632a00700 1 -- 172.21.15.27:0/1682670819 <== mon.1 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f362800e040 con 0x7f363c0fe0e0 2024-04-16T05:43:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.045+0000 7f3632a00700 1 -- 172.21.15.27:0/1682670819 <== mon.1 v2:172.21.15.122:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f362800bbf0 con 0x7f363c0fe0e0 2024-04-16T05:43:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.045+0000 7f3632a00700 1 -- 172.21.15.27:0/1682670819 <== mon.1 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f362801c630 con 0x7f363c0fe0e0 2024-04-16T05:43:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.045+0000 7f3641800700 1 -- 172.21.15.27:0/1682670819 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f363c111270 con 0x7f363c0fe0e0 2024-04-16T05:43:11.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.045+0000 7f3641800700 1 -- 172.21.15.27:0/1682670819 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f363c1117b0 con 0x7f363c0fe0e0 2024-04-16T05:43:11.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.046+0000 7f3641800700 1 -- 172.21.15.27:0/1682670819 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f363c101a70 con 0x7f363c0fe0e0 2024-04-16T05:43:11.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.048+0000 7f3632a00700 1 -- 172.21.15.27:0/1682670819 <== mon.1 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 25) v1 ==== 89652+0+0 (secure 0 0 0) 0x7f362800bd60 con 0x7f363c0fe0e0 2024-04-16T05:43:11.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.049+0000 7f3632a00700 1 --2- 172.21.15.27:0/1682670819 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f362006bd40 0x7f362006e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:11.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.049+0000 7f3632a00700 1 -- 172.21.15.27:0/1682670819 <== mon.1 v2:172.21.15.122:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f362808ca40 con 0x7f363c0fe0e0 2024-04-16T05:43:11.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.050+0000 7f363aa00700 1 --2- 172.21.15.27:0/1682670819 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f362006bd40 0x7f362006e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:11.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.051+0000 7f363aa00700 1 --2- 172.21.15.27:0/1682670819 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f362006bd40 0x7f362006e1f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f362c009fd0 tx=0x7f362c009f00 comp rx=0 tx=0).ready entity=mgr.24369 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:11.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.065+0000 7f3632a00700 1 -- 172.21.15.27:0/1682670819 <== mon.1 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f3628059b30 con 0x7f363c0fe0e0 2024-04-16T05:43:11.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:10 smithi122 ceph-mon[35998]: Standby manager daemon smithi027.qgzypi started 2024-04-16T05:43:11.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:10 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.27:0/201953492' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.qgzypi/crt"}]: dispatch 2024-04-16T05:43:11.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:10 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.27:0/201953492' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T05:43:11.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:10 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.27:0/201953492' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.qgzypi/key"}]: dispatch 2024-04-16T05:43:11.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:10 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.27:0/201953492' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T05:43:11.340 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.341+0000 7f3641800700 1 -- 172.21.15.27:0/1682670819 --> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f363c111a90 con 0x7f362006bd40 2024-04-16T05:43:11.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.344+0000 7f3632a00700 1 -- 172.21.15.27:0/1682670819 <== mgr.24369 v2:172.21.15.122:6828/871482489 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7f363c111a90 con 0x7f362006bd40 2024-04-16T05:43:11.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.347+0000 7f3641800700 1 -- 172.21.15.27:0/1682670819 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f362006bd40 msgr2=0x7f362006e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:11.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.347+0000 7f3641800700 1 --2- 172.21.15.27:0/1682670819 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f362006bd40 0x7f362006e1f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f362c009fd0 tx=0x7f362c009f00 comp rx=0 tx=0).stop 2024-04-16T05:43:11.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.347+0000 7f3641800700 1 -- 172.21.15.27:0/1682670819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f363c0fe0e0 msgr2=0x7f363c1101a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.347+0000 7f3641800700 1 --2- 172.21.15.27:0/1682670819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f363c0fe0e0 0x7f363c1101a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3628009150 tx=0x7f3628009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.347+0000 7f3641800700 1 -- 172.21.15.27:0/1682670819 shutdown_connections 2024-04-16T05:43:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.347+0000 7f3641800700 1 --2- 172.21.15.27:0/1682670819 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f362006bd40 0x7f362006e1f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.347+0000 7f3641800700 1 --2- 172.21.15.27:0/1682670819 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f363c0fcef0 0x7f363c10fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.347+0000 7f3641800700 1 --2- 172.21.15.27:0/1682670819 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f363c0fe0e0 0x7f363c1101a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.347+0000 7f3641800700 1 -- 172.21.15.27:0/1682670819 >> 172.21.15.27:0/1682670819 conn(0x7f363c0f8530 msgr2=0x7f363c0f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.347+0000 7f3641800700 1 -- 172.21.15.27:0/1682670819 shutdown_connections 2024-04-16T05:43:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.347+0000 7f3641800700 1 -- 172.21.15.27:0/1682670819 wait complete. 2024-04-16T05:43:11.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.501+0000 7f47a9200700 1 -- 172.21.15.27:0/2322097640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47a40ff020 msgr2=0x7f47a40ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:11.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.501+0000 7f47a9200700 1 --2- 172.21.15.27:0/2322097640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47a40ff020 0x7f47a40ff430 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f4794004940 tx=0x7f4794009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:11.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.502+0000 7f47a9200700 1 -- 172.21.15.27:0/2322097640 shutdown_connections 2024-04-16T05:43:11.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.502+0000 7f47a9200700 1 --2- 172.21.15.27:0/2322097640 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47a41002d0 0x7f47a4100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:11.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.502+0000 7f47a9200700 1 --2- 172.21.15.27:0/2322097640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47a40ff020 0x7f47a40ff430 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:11.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.502+0000 7f47a9200700 1 -- 172.21.15.27:0/2322097640 >> 172.21.15.27:0/2322097640 conn(0x7f47a40fa950 msgr2=0x7f47a40fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:11.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.502+0000 7f47a9200700 1 -- 172.21.15.27:0/2322097640 shutdown_connections 2024-04-16T05:43:11.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.502+0000 7f47a9200700 1 -- 172.21.15.27:0/2322097640 wait complete. 2024-04-16T05:43:11.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.503+0000 7f47a9200700 1 Processor -- start 2024-04-16T05:43:11.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.503+0000 7f47a9200700 1 -- start start 2024-04-16T05:43:11.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.503+0000 7f47a9200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47a40ff020 0x7f47a410fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:11.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.503+0000 7f47a9200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47a41002d0 0x7f47a41100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:11.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.503+0000 7f47a9200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a41106a0 con 0x7f47a41002d0 2024-04-16T05:43:11.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.503+0000 7f47a9200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a4110810 con 0x7f47a40ff020 2024-04-16T05:43:11.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.503+0000 7f47a1400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47a41002d0 0x7f47a41100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:11.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.504+0000 7f47a1400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47a41002d0 0x7f47a41100a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57900/0 (socket says 172.21.15.27:57900) 2024-04-16T05:43:11.503 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.504+0000 7f47a1400700 1 -- 172.21.15.27:0/276741349 learned_addr learned my addr 172.21.15.27:0/276741349 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:43:11.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.504+0000 7f47a1e00700 1 --2- 172.21.15.27:0/276741349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47a40ff020 0x7f47a410fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:11.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.504+0000 7f47a1400700 1 -- 172.21.15.27:0/276741349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47a40ff020 msgr2=0x7f47a410fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:11.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.504+0000 7f47a1400700 1 --2- 172.21.15.27:0/276741349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47a40ff020 0x7f47a410fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:11.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.504+0000 7f47a1400700 1 -- 172.21.15.27:0/276741349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4794005040 con 0x7f47a41002d0 2024-04-16T05:43:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.504+0000 7f47a1400700 1 --2- 172.21.15.27:0/276741349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47a41002d0 0x7f47a41100a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4790002a50 tx=0x7f47900046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.505+0000 7f479a000700 1 -- 172.21.15.27:0/276741349 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4790007cc0 con 0x7f47a41002d0 2024-04-16T05:43:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.505+0000 7f479a000700 1 -- 172.21.15.27:0/276741349 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f479000f040 con 0x7f47a41002d0 2024-04-16T05:43:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.505+0000 7f479a000700 1 -- 172.21.15.27:0/276741349 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4790009640 con 0x7f47a41002d0 2024-04-16T05:43:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.505+0000 7f47a9200700 1 -- 172.21.15.27:0/276741349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a4115270 con 0x7f47a41002d0 2024-04-16T05:43:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.505+0000 7f47a9200700 1 -- 172.21.15.27:0/276741349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a4115830 con 0x7f47a41002d0 2024-04-16T05:43:11.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.506+0000 7f47a9200700 1 -- 172.21.15.27:0/276741349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a404c710 con 0x7f47a41002d0 2024-04-16T05:43:11.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.507+0000 7f479a000700 1 -- 172.21.15.27:0/276741349 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 89652+0+0 (secure 0 0 0) 0x7f47900097a0 con 0x7f47a41002d0 2024-04-16T05:43:11.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.508+0000 7f479a000700 1 --2- 172.21.15.27:0/276741349 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f478806bd00 0x7f478806e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:11.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.508+0000 7f479a000700 1 -- 172.21.15.27:0/276741349 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f479008a880 con 0x7f47a41002d0 2024-04-16T05:43:11.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.509+0000 7f47a1e00700 1 --2- 172.21.15.27:0/276741349 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f478806bd00 0x7f478806e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:11.509 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.510+0000 7f47a1e00700 1 --2- 172.21.15.27:0/276741349 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f478806bd00 0x7f478806e1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f479400b3b0 tx=0x7f4794009e30 comp rx=0 tx=0).ready entity=mgr.24369 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:11.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.513+0000 7f479a000700 1 -- 172.21.15.27:0/276741349 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f47900578f0 con 0x7f47a41002d0 2024-04-16T05:43:11.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.763+0000 7f47a9200700 1 -- 172.21.15.27:0/276741349 --> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f47a4104640 con 0x7f478806bd00 2024-04-16T05:43:11.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.801+0000 7f479a000700 1 -- 172.21.15.27:0/276741349 <== mgr.24369 v2:172.21.15.122:6828/871482489 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7f47a4104640 con 0x7f478806bd00 2024-04-16T05:43:11.801 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:43:11.801 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (8m) 2m ago 10m 13.4M - 0.25.0 c8568f914cd2 95a43269c2c6 2024-04-16T05:43:11.801 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (10m) 2m ago 10m 6665k - 17.2.7-847-g45317f95 b333581ccdf5 f1acb6959c63 2024-04-16T05:43:11.801 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (9m) 18s ago 9m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 e81efc20e8c0 2024-04-16T05:43:11.801 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (8m) 2m ago 9m 73.3M - 9.4.7 954c08fa6188 0de7ee73579d 2024-04-16T05:43:11.801 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (2m) 2m ago 2m 12.8M - 17.2.7-847-g45317f95 b333581ccdf5 0824e2621be5 2024-04-16T05:43:11.801 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (2m) 2m ago 2m 10.5M - 17.2.7-847-g45317f95 b333581ccdf5 b332a72cc594 2024-04-16T05:43:11.801 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (2m) 18s ago 2m 11.2M - 17.2.7-847-g45317f95 b333581ccdf5 f5e36815ddfa 2024-04-16T05:43:11.801 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (2m) 18s ago 2m 11.2M - 17.2.7-847-g45317f95 b333581ccdf5 755a73c65a7f 2024-04-16T05:43:11.801 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:9283 running (12m) 2m ago 12m 481M - 17.2.7-847-g45317f95 b333581ccdf5 78f461c198fe 2024-04-16T05:43:11.801 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283 running (22s) 18s ago 8m 59.3M - 19.0.0-2884-g218a6914 a4e4811815e4 b1a1b7a6d536 2024-04-16T05:43:11.801 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (12m) 2m ago 12m 44.8M 2048M 17.2.7-847-g45317f95 b333581ccdf5 88575dbb8d98 2024-04-16T05:43:11.802 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (8m) 18s ago 8m 32.4M 2048M 17.2.7-847-g45317f95 b333581ccdf5 5df19eab6d21 2024-04-16T05:43:11.802 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (10m) 2m ago 10m 7763k - 1.5.0 0da6a335fe13 26bcc8d81c89 2024-04-16T05:43:11.802 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (8m) 18s ago 8m 7524k - 1.5.0 0da6a335fe13 7f796c66dff6 2024-04-16T05:43:11.802 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (7m) 2m ago 7m 44.8M 4288M 17.2.7-847-g45317f95 b333581ccdf5 15e049573bca 2024-04-16T05:43:11.802 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (7m) 2m ago 7m 45.1M 4288M 17.2.7-847-g45317f95 b333581ccdf5 e146295f6500 2024-04-16T05:43:11.802 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (6m) 2m ago 6m 46.0M 4288M 17.2.7-847-g45317f95 b333581ccdf5 6212f6a18fa4 2024-04-16T05:43:11.802 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (5m) 18s ago 5m 44.0M 5312M 17.2.7-847-g45317f95 b333581ccdf5 d6724ad05f9f 2024-04-16T05:43:11.802 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (5m) 18s ago 5m 42.4M 5312M 17.2.7-847-g45317f95 b333581ccdf5 7a59908a3e10 2024-04-16T05:43:11.802 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (4m) 18s ago 4m 41.7M 5312M 17.2.7-847-g45317f95 b333581ccdf5 8e4b60163907 2024-04-16T05:43:11.802 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (7m) 2m ago 9m 46.8M - 2.43.0 a07b618ecd1d 719577e1340d 2024-04-16T05:43:11.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.804+0000 7f47a9200700 1 -- 172.21.15.27:0/276741349 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f478806bd00 msgr2=0x7f478806e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:11.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.804+0000 7f47a9200700 1 --2- 172.21.15.27:0/276741349 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f478806bd00 0x7f478806e1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f479400b3b0 tx=0x7f4794009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:11.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.804+0000 7f47a9200700 1 -- 172.21.15.27:0/276741349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47a41002d0 msgr2=0x7f47a41100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:11.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.804+0000 7f47a9200700 1 --2- 172.21.15.27:0/276741349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47a41002d0 0x7f47a41100a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4790002a50 tx=0x7f47900046b0 comp rx=0 tx=0).stop 2024-04-16T05:43:11.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.805+0000 7f47a9200700 1 -- 172.21.15.27:0/276741349 shutdown_connections 2024-04-16T05:43:11.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.805+0000 7f47a9200700 1 --2- 172.21.15.27:0/276741349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f47a40ff020 0x7f47a410fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:11.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.805+0000 7f47a9200700 1 --2- 172.21.15.27:0/276741349 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f478806bd00 0x7f478806e1b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:11.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.805+0000 7f47a9200700 1 --2- 172.21.15.27:0/276741349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47a41002d0 0x7f47a41100a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:11.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.805+0000 7f47a9200700 1 -- 172.21.15.27:0/276741349 >> 172.21.15.27:0/276741349 conn(0x7f47a40fa950 msgr2=0x7f47a40fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:11.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.805+0000 7f47a9200700 1 -- 172.21.15.27:0/276741349 shutdown_connections 2024-04-16T05:43:11.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.805+0000 7f47a9200700 1 -- 172.21.15.27:0/276741349 wait complete. 2024-04-16T05:43:11.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.961+0000 7fcde8800700 1 -- 172.21.15.27:0/2144372093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde01046d0 msgr2=0x7fcde0104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:11.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.961+0000 7fcde8800700 1 --2- 172.21.15.27:0/2144372093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde01046d0 0x7fcde0104b40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0004940 tx=0x7fcdd0009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:11.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.961+0000 7fcde8800700 1 -- 172.21.15.27:0/2144372093 shutdown_connections 2024-04-16T05:43:11.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.961+0000 7fcde8800700 1 --2- 172.21.15.27:0/2144372093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde01046d0 0x7fcde0104b40 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:11.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.961+0000 7fcde8800700 1 --2- 172.21.15.27:0/2144372093 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcde0103420 0x7fcde0103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:11.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.961+0000 7fcde8800700 1 -- 172.21.15.27:0/2144372093 >> 172.21.15.27:0/2144372093 conn(0x7fcde0072640 msgr2=0x7fcde0074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:11.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.962+0000 7fcde8800700 1 -- 172.21.15.27:0/2144372093 shutdown_connections 2024-04-16T05:43:11.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.962+0000 7fcde8800700 1 -- 172.21.15.27:0/2144372093 wait complete. 2024-04-16T05:43:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.962+0000 7fcde8800700 1 Processor -- start 2024-04-16T05:43:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.963+0000 7fcde8800700 1 -- start start 2024-04-16T05:43:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.963+0000 7fcde8800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde0103420 0x7fcde0113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.963+0000 7fcde8800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcde01046d0 0x7fcde0114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.963+0000 7fcde8800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde0114b20 con 0x7fcde0103420 2024-04-16T05:43:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.963+0000 7fcde8800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde01194c0 con 0x7fcde01046d0 2024-04-16T05:43:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.963+0000 7fcde5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde0103420 0x7fcde0113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.963+0000 7fcde5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde0103420 0x7fcde0113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57918/0 (socket says 172.21.15.27:57918) 2024-04-16T05:43:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.963+0000 7fcde5e00700 1 -- 172.21.15.27:0/3562306532 learned_addr learned my addr 172.21.15.27:0/3562306532 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:43:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.964+0000 7fcde5400700 1 --2- 172.21.15.27:0/3562306532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcde01046d0 0x7fcde0114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.964+0000 7fcde5e00700 1 -- 172.21.15.27:0/3562306532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcde01046d0 msgr2=0x7fcde0114490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.964+0000 7fcde5e00700 1 --2- 172.21.15.27:0/3562306532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcde01046d0 0x7fcde0114490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.964+0000 7fcde5e00700 1 -- 172.21.15.27:0/3562306532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd0005040 con 0x7fcde0103420 2024-04-16T05:43:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.964+0000 7fcde5e00700 1 --2- 172.21.15.27:0/3562306532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde0103420 0x7fcde0113f50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400b790 tx=0x7fcdd4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.964+0000 7fcdda000700 1 -- 172.21.15.27:0/3562306532 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdd400f770 con 0x7fcde0103420 2024-04-16T05:43:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.964+0000 7fcdda000700 1 -- 172.21.15.27:0/3562306532 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcdd400fda0 con 0x7fcde0103420 2024-04-16T05:43:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.964+0000 7fcde8800700 1 -- 172.21.15.27:0/3562306532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde0119720 con 0x7fcde0103420 2024-04-16T05:43:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.964+0000 7fcdda000700 1 -- 172.21.15.27:0/3562306532 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdd400d310 con 0x7fcde0103420 2024-04-16T05:43:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.965+0000 7fcde8800700 1 -- 172.21.15.27:0/3562306532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde0119c90 con 0x7fcde0103420 2024-04-16T05:43:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.966+0000 7fcde8800700 1 -- 172.21.15.27:0/3562306532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcde004c710 con 0x7fcde0103420 2024-04-16T05:43:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.967+0000 7fcdda000700 1 -- 172.21.15.27:0/3562306532 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 89652+0+0 (secure 0 0 0) 0x7fcdd4010420 con 0x7fcde0103420 2024-04-16T05:43:11.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.968+0000 7fcdda000700 1 --2- 172.21.15.27:0/3562306532 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fcdc806bd00 0x7fcdc806e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:11.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.968+0000 7fcdda000700 1 -- 172.21.15.27:0/3562306532 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fcdd408a980 con 0x7fcde0103420 2024-04-16T05:43:11.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.968+0000 7fcde5400700 1 --2- 172.21.15.27:0/3562306532 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fcdc806bd00 0x7fcdc806e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:11.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.969+0000 7fcde5400700 1 --2- 172.21.15.27:0/3562306532 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fcdc806bd00 0x7fcdc806e1b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0004f90 tx=0x7fcdd0009e30 comp rx=0 tx=0).ready entity=mgr.24369 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:11.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:11.972+0000 7fcdda000700 1 -- 172.21.15.27:0/3562306532 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fcdd4057b80 con 0x7fcde0103420 2024-04-16T05:43:12.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:11 smithi122 ceph-mon[35998]: pgmap v8: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-16T05:43:12.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:11 smithi122 ceph-mon[35998]: mgrmap e25: smithi122.sgokkq(active, since 10s), standbys: smithi027.qgzypi 2024-04-16T05:43:12.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:11 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mgr metadata", "who": "smithi027.qgzypi", "id": "smithi027.qgzypi"}]: dispatch 2024-04-16T05:43:12.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:11 smithi122 ceph-mon[35998]: from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:43:12.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:11 smithi122 ceph-mon[35998]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:43:12.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:11 smithi027 ceph-mon[29206]: pgmap v8: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-16T05:43:12.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:11 smithi027 ceph-mon[29206]: mgrmap e25: smithi122.sgokkq(active, since 10s), standbys: smithi027.qgzypi 2024-04-16T05:43:12.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:11 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mgr metadata", "who": "smithi027.qgzypi", "id": "smithi027.qgzypi"}]: dispatch 2024-04-16T05:43:12.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:11 smithi027 ceph-mon[29206]: from='client.14660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:43:12.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:11 smithi027 ceph-mon[29206]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:43:12.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.319+0000 7fcde8800700 1 -- 172.21.15.27:0/3562306532 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcde011a1c0 con 0x7fcde0103420 2024-04-16T05:43:12.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.320+0000 7fcdda000700 1 -- 172.21.15.27:0/3562306532 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+846 (secure 0 0 0) 0x7fcdd4057710 con 0x7fcde0103420 2024-04-16T05:43:12.319 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:43:12.319 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:43:12.319 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T05:43:12.319 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:43:12.319 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:43:12.319 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 1, 2024-04-16T05:43:12.320 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 1 2024-04-16T05:43:12.320 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:43:12.320 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:43:12.320 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T05:43:12.320 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:43:12.320 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:43:12.320 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T05:43:12.320 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:43:12.320 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:43:12.320 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 13, 2024-04-16T05:43:12.320 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 1 2024-04-16T05:43:12.320 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:43:12.320 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:43:12.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.323+0000 7fcde8800700 1 -- 172.21.15.27:0/3562306532 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fcdc806bd00 msgr2=0x7fcdc806e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:12.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.323+0000 7fcde8800700 1 --2- 172.21.15.27:0/3562306532 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fcdc806bd00 0x7fcdc806e1b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0004f90 tx=0x7fcdd0009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:12.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.323+0000 7fcde8800700 1 -- 172.21.15.27:0/3562306532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde0103420 msgr2=0x7fcde0113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:12.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.323+0000 7fcde8800700 1 --2- 172.21.15.27:0/3562306532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde0103420 0x7fcde0113f50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400b790 tx=0x7fcdd4009250 comp rx=0 tx=0).stop 2024-04-16T05:43:12.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.323+0000 7fcde8800700 1 -- 172.21.15.27:0/3562306532 shutdown_connections 2024-04-16T05:43:12.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.323+0000 7fcde8800700 1 --2- 172.21.15.27:0/3562306532 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fcdc806bd00 0x7fcdc806e1b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:12.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.323+0000 7fcde8800700 1 --2- 172.21.15.27:0/3562306532 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcde0103420 0x7fcde0113f50 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:12.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.323+0000 7fcde8800700 1 --2- 172.21.15.27:0/3562306532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcde01046d0 0x7fcde0114490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:12.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.323+0000 7fcde8800700 1 -- 172.21.15.27:0/3562306532 >> 172.21.15.27:0/3562306532 conn(0x7fcde0072640 msgr2=0x7fcde0073ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:12.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.323+0000 7fcde8800700 1 -- 172.21.15.27:0/3562306532 shutdown_connections 2024-04-16T05:43:12.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.323+0000 7fcde8800700 1 -- 172.21.15.27:0/3562306532 wait complete. 2024-04-16T05:43:12.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.482+0000 7fbd55600700 1 -- 172.21.15.27:0/3092127472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd50066b20 msgr2=0x7fbd500ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:12.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.482+0000 7fbd55600700 1 --2- 172.21.15.27:0/3092127472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd50066b20 0x7fbd500ffa40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fbd40004940 tx=0x7fbd40009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:12.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.482+0000 7fbd55600700 1 -- 172.21.15.27:0/3092127472 shutdown_connections 2024-04-16T05:43:12.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.482+0000 7fbd55600700 1 --2- 172.21.15.27:0/3092127472 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd500fff80 0x7fbd50102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:12.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.482+0000 7fbd55600700 1 --2- 172.21.15.27:0/3092127472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd50066b20 0x7fbd500ffa40 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:12.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.482+0000 7fbd55600700 1 -- 172.21.15.27:0/3092127472 >> 172.21.15.27:0/3092127472 conn(0x7fbd500f7450 msgr2=0x7fbd500f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:12.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.482+0000 7fbd55600700 1 -- 172.21.15.27:0/3092127472 shutdown_connections 2024-04-16T05:43:12.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.482+0000 7fbd55600700 1 -- 172.21.15.27:0/3092127472 wait complete. 2024-04-16T05:43:12.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.483+0000 7fbd55600700 1 Processor -- start 2024-04-16T05:43:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.483+0000 7fbd55600700 1 -- start start 2024-04-16T05:43:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.483+0000 7fbd55600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd50066b20 0x7fbd5010fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.483+0000 7fbd55600700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd500fff80 0x7fbd50110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.484+0000 7fbd55600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd50110850 con 0x7fbd50066b20 2024-04-16T05:43:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.484+0000 7fbd55600700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd50110f90 con 0x7fbd500fff80 2024-04-16T05:43:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.484+0000 7fbd4ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd50066b20 0x7fbd5010fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.484+0000 7fbd4ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd50066b20 0x7fbd5010fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57934/0 (socket says 172.21.15.27:57934) 2024-04-16T05:43:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.484+0000 7fbd4ea00700 1 -- 172.21.15.27:0/1463316226 learned_addr learned my addr 172.21.15.27:0/1463316226 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:43:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.484+0000 7fbd4e000700 1 --2- 172.21.15.27:0/1463316226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd500fff80 0x7fbd50110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.484+0000 7fbd4ea00700 1 -- 172.21.15.27:0/1463316226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd500fff80 msgr2=0x7fbd50110250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.484+0000 7fbd4ea00700 1 --2- 172.21.15.27:0/1463316226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd500fff80 0x7fbd50110250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.484+0000 7fbd4ea00700 1 -- 172.21.15.27:0/1463316226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd40005040 con 0x7fbd50066b20 2024-04-16T05:43:12.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.485+0000 7fbd4ea00700 1 --2- 172.21.15.27:0/1463316226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd50066b20 0x7fbd5010fd10 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbd4000b970 tx=0x7fbd40009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:12.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.485+0000 7fbd46a00700 1 -- 172.21.15.27:0/1463316226 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd4000e040 con 0x7fbd50066b20 2024-04-16T05:43:12.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.485+0000 7fbd46a00700 1 -- 172.21.15.27:0/1463316226 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd4000bbf0 con 0x7fbd50066b20 2024-04-16T05:43:12.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.485+0000 7fbd55600700 1 -- 172.21.15.27:0/1463316226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd50111210 con 0x7fbd50066b20 2024-04-16T05:43:12.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.485+0000 7fbd46a00700 1 -- 172.21.15.27:0/1463316226 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd4001c630 con 0x7fbd50066b20 2024-04-16T05:43:12.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.485+0000 7fbd55600700 1 -- 172.21.15.27:0/1463316226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd501117d0 con 0x7fbd50066b20 2024-04-16T05:43:12.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.487+0000 7fbd55600700 1 -- 172.21.15.27:0/1463316226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5005f5f0 con 0x7fbd50066b20 2024-04-16T05:43:12.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.488+0000 7fbd46a00700 1 -- 172.21.15.27:0/1463316226 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 89652+0+0 (secure 0 0 0) 0x7fbd4000bd60 con 0x7fbd50066b20 2024-04-16T05:43:12.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.489+0000 7fbd46a00700 1 --2- 172.21.15.27:0/1463316226 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fbd3406bcf0 0x7fbd3406e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:12.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.489+0000 7fbd46a00700 1 -- 172.21.15.27:0/1463316226 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fbd4008be70 con 0x7fbd50066b20 2024-04-16T05:43:12.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.490+0000 7fbd4e000700 1 --2- 172.21.15.27:0/1463316226 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fbd3406bcf0 0x7fbd3406e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:12.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.491+0000 7fbd4e000700 1 --2- 172.21.15.27:0/1463316226 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fbd3406bcf0 0x7fbd3406e1a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c007fd0 tx=0x7fbd3c009040 comp rx=0 tx=0).ready entity=mgr.24369 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:12.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.494+0000 7fbd46a00700 1 -- 172.21.15.27:0/1463316226 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fbd40059010 con 0x7fbd50066b20 2024-04-16T05:43:12.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.767+0000 7fbd55600700 1 -- 172.21.15.27:0/1463316226 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbd501058d0 con 0x7fbd50066b20 2024-04-16T05:43:12.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.768+0000 7fbd46a00700 1 -- 172.21.15.27:0/1463316226 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1810 (secure 0 0 0) 0x7fbd40058ba0 con 0x7fbd50066b20 2024-04-16T05:43:12.767 INFO:teuthology.orchestra.run.smithi027.stdout:e16 2024-04-16T05:43:12.767 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:43:12.767 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:43:12.767 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:43:12.767 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:43:12.768 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:43:12.768 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:43:12.768 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 15 2024-04-16T05:43:12.768 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:43:12.768 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:43:12.768 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:40:32.701649+0000 2024-04-16T05:43:12.768 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:43:12.768 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:43:12.768 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:43:12.768 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:43:12.768 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:43:12.768 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:43:12.768 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:43:12.768 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 46 2024-04-16T05:43:12.768 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:43:12.769 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:43:12.769 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:43:12.769 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=14462} 2024-04-16T05:43:12.769 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:43:12.769 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:43:12.769 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:43:12.769 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:43:12.769 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:43:12.769 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:43:12.769 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:43:12.769 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:43:12.769 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:43:12.769 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:43:12.769 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:43:12.770 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:43:12.770 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:43:12.770 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{-1:14474} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2065690722,v1:172.21.15.27:6827/2065690722] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:43:12.770 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:24263} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:43:12.770 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:24273} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:43:12.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.770+0000 7fbd55600700 1 -- 172.21.15.27:0/1463316226 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fbd3406bcf0 msgr2=0x7fbd3406e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:12.770 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.770+0000 7fbd55600700 1 --2- 172.21.15.27:0/1463316226 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fbd3406bcf0 0x7fbd3406e1a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c007fd0 tx=0x7fbd3c009040 comp rx=0 tx=0).stop 2024-04-16T05:43:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.770+0000 7fbd55600700 1 -- 172.21.15.27:0/1463316226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd50066b20 msgr2=0x7fbd5010fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.770+0000 7fbd55600700 1 --2- 172.21.15.27:0/1463316226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd50066b20 0x7fbd5010fd10 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbd4000b970 tx=0x7fbd40009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.770+0000 7fbd55600700 1 -- 172.21.15.27:0/1463316226 shutdown_connections 2024-04-16T05:43:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.770+0000 7fbd55600700 1 --2- 172.21.15.27:0/1463316226 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fbd3406bcf0 0x7fbd3406e1a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.770+0000 7fbd55600700 1 --2- 172.21.15.27:0/1463316226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd50066b20 0x7fbd5010fd10 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.770+0000 7fbd55600700 1 --2- 172.21.15.27:0/1463316226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd500fff80 0x7fbd50110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.770+0000 7fbd55600700 1 -- 172.21.15.27:0/1463316226 >> 172.21.15.27:0/1463316226 conn(0x7fbd500f7450 msgr2=0x7fbd500fba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.770+0000 7fbd55600700 1 -- 172.21.15.27:0/1463316226 shutdown_connections 2024-04-16T05:43:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.770+0000 7fbd55600700 1 -- 172.21.15.27:0/1463316226 wait complete. 2024-04-16T05:43:12.771 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2024-04-16T05:43:12.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.940+0000 7f733ce00700 1 -- 172.21.15.27:0/2125858562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73380708f0 msgr2=0x7f7338070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:12.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.940+0000 7f733ce00700 1 --2- 172.21.15.27:0/2125858562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73380708f0 0x7f7338070d60 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7324004940 tx=0x7f7324009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:12.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.941+0000 7f733ce00700 1 -- 172.21.15.27:0/2125858562 shutdown_connections 2024-04-16T05:43:12.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.941+0000 7f733ce00700 1 --2- 172.21.15.27:0/2125858562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f73380708f0 0x7f7338070d60 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:12.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.941+0000 7f733ce00700 1 --2- 172.21.15.27:0/2125858562 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7338071ea0 0x7f73380703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:12.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.941+0000 7f733ce00700 1 -- 172.21.15.27:0/2125858562 >> 172.21.15.27:0/2125858562 conn(0x7f73380f8580 msgr2=0x7f73380fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:12.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.941+0000 7f733ce00700 1 -- 172.21.15.27:0/2125858562 shutdown_connections 2024-04-16T05:43:12.940 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.941+0000 7f733ce00700 1 -- 172.21.15.27:0/2125858562 wait complete. 2024-04-16T05:43:12.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.942+0000 7f733ce00700 1 Processor -- start 2024-04-16T05:43:12.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.942+0000 7f733ce00700 1 -- start start 2024-04-16T05:43:12.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.942+0000 7f733ce00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73380708f0 0x7f7338113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:12.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.942+0000 7f733ce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7338071ea0 0x7f7338114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:12.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.942+0000 7f733ce00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7338114a90 con 0x7f7338071ea0 2024-04-16T05:43:12.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.942+0000 7f733ce00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7338114c00 con 0x7f73380708f0 2024-04-16T05:43:12.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.943+0000 7f7335400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7338071ea0 0x7f7338114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:12.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.943+0000 7f7335400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7338071ea0 0x7f7338114490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57960/0 (socket says 172.21.15.27:57960) 2024-04-16T05:43:12.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.943+0000 7f7335400700 1 -- 172.21.15.27:0/2464126017 learned_addr learned my addr 172.21.15.27:0/2464126017 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:43:12.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.943+0000 7f7335e00700 1 --2- 172.21.15.27:0/2464126017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73380708f0 0x7f7338113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:12.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.943+0000 7f7335400700 1 -- 172.21.15.27:0/2464126017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73380708f0 msgr2=0x7f7338113f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:12.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.943+0000 7f7335400700 1 --2- 172.21.15.27:0/2464126017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73380708f0 0x7f7338113f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:12.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.943+0000 7f7335400700 1 -- 172.21.15.27:0/2464126017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7324005040 con 0x7f7338071ea0 2024-04-16T05:43:12.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.943+0000 7f7335e00700 1 --2- 172.21.15.27:0/2464126017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73380708f0 0x7f7338113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T05:43:12.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.944+0000 7f7335400700 1 --2- 172.21.15.27:0/2464126017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7338071ea0 0x7f7338114490 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f732400b3b0 tx=0x7f7324009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:12.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.944+0000 7f732e000700 1 -- 172.21.15.27:0/2464126017 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f732400e040 con 0x7f7338071ea0 2024-04-16T05:43:12.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.944+0000 7f732e000700 1 -- 172.21.15.27:0/2464126017 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7324004290 con 0x7f7338071ea0 2024-04-16T05:43:12.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.944+0000 7f732e000700 1 -- 172.21.15.27:0/2464126017 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f732401c630 con 0x7f7338071ea0 2024-04-16T05:43:12.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.944+0000 7f733ce00700 1 -- 172.21.15.27:0/2464126017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7338119600 con 0x7f7338071ea0 2024-04-16T05:43:12.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.944+0000 7f733ce00700 1 -- 172.21.15.27:0/2464126017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7338119bc0 con 0x7f7338071ea0 2024-04-16T05:43:12.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.946+0000 7f733ce00700 1 -- 172.21.15.27:0/2464126017 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f733804c710 con 0x7f7338071ea0 2024-04-16T05:43:12.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.947+0000 7f732e000700 1 -- 172.21.15.27:0/2464126017 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 89652+0+0 (secure 0 0 0) 0x7f732400bbf0 con 0x7f7338071ea0 2024-04-16T05:43:12.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.948+0000 7f732e000700 1 --2- 172.21.15.27:0/2464126017 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f731c06bd00 0x7f731c06e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:12.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.948+0000 7f732e000700 1 -- 172.21.15.27:0/2464126017 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f732408bf30 con 0x7f7338071ea0 2024-04-16T05:43:12.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.949+0000 7f7335e00700 1 --2- 172.21.15.27:0/2464126017 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f731c06bd00 0x7f731c06e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:12.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.950+0000 7f7335e00700 1 --2- 172.21.15.27:0/2464126017 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f731c06bd00 0x7f731c06e1b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7328009fd0 tx=0x7f7328009f00 comp rx=0 tx=0).ready entity=mgr.24369 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:12.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:12.953+0000 7f732e000700 1 -- 172.21.15.27:0/2464126017 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f7324059020 con 0x7f7338071ea0 2024-04-16T05:43:13.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:12 smithi027 ceph-mon[29206]: from='client.14668 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:43:13.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:12 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/3562306532' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:43:13.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:12 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1463316226' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:43:13.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.259+0000 7f733ce00700 1 -- 172.21.15.27:0/2464126017 --> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f73380ff7c0 con 0x7f731c06bd00 2024-04-16T05:43:13.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.263+0000 7f732e000700 1 -- 172.21.15.27:0/2464126017 <== mgr.24369 v2:172.21.15.122:6828/871482489 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7f73380ff7c0 con 0x7f731c06bd00 2024-04-16T05:43:13.262 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:43:13.263 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T05:43:13.263 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:43:13.263 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:43:13.263 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [], 2024-04-16T05:43:13.263 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "1/21 daemons upgraded", 2024-04-16T05:43:13.263 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "", 2024-04-16T05:43:13.263 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:43:13.263 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:43:13.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.266+0000 7f733ce00700 1 -- 172.21.15.27:0/2464126017 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f731c06bd00 msgr2=0x7f731c06e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:13.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.266+0000 7f733ce00700 1 --2- 172.21.15.27:0/2464126017 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f731c06bd00 0x7f731c06e1b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7328009fd0 tx=0x7f7328009f00 comp rx=0 tx=0).stop 2024-04-16T05:43:13.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.266+0000 7f733ce00700 1 -- 172.21.15.27:0/2464126017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7338071ea0 msgr2=0x7f7338114490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:13.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.266+0000 7f733ce00700 1 --2- 172.21.15.27:0/2464126017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7338071ea0 0x7f7338114490 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f732400b3b0 tx=0x7f7324009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:13.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.266+0000 7f733ce00700 1 -- 172.21.15.27:0/2464126017 shutdown_connections 2024-04-16T05:43:13.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.266+0000 7f733ce00700 1 --2- 172.21.15.27:0/2464126017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73380708f0 0x7f7338113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:13.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.266+0000 7f733ce00700 1 --2- 172.21.15.27:0/2464126017 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f731c06bd00 0x7f731c06e1b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:13.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.266+0000 7f733ce00700 1 --2- 172.21.15.27:0/2464126017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7338071ea0 0x7f7338114490 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:13.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.266+0000 7f733ce00700 1 -- 172.21.15.27:0/2464126017 >> 172.21.15.27:0/2464126017 conn(0x7f73380f8580 msgr2=0x7f73380fa0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:13.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.266+0000 7f733ce00700 1 -- 172.21.15.27:0/2464126017 shutdown_connections 2024-04-16T05:43:13.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.266+0000 7f733ce00700 1 -- 172.21.15.27:0/2464126017 wait complete. 2024-04-16T05:43:13.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:12 smithi122 ceph-mon[35998]: from='client.14668 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:43:13.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:12 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/3562306532' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:43:13.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:12 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/1463316226' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:43:13.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.421+0000 7efe0c000700 1 -- 172.21.15.27:0/2913786841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe040fce40 msgr2=0x7efe040fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:13.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.421+0000 7efe0c000700 1 --2- 172.21.15.27:0/2913786841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe040fce40 0x7efe040fd250 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7efdf8004940 tx=0x7efdf8009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:13.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.421+0000 7efe0c000700 1 -- 172.21.15.27:0/2913786841 shutdown_connections 2024-04-16T05:43:13.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.421+0000 7efe0c000700 1 --2- 172.21.15.27:0/2913786841 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe040fe040 0x7efe040fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:13.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.421+0000 7efe0c000700 1 --2- 172.21.15.27:0/2913786841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe040fce40 0x7efe040fd250 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:13.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.421+0000 7efe0c000700 1 -- 172.21.15.27:0/2913786841 >> 172.21.15.27:0/2913786841 conn(0x7efe040f8530 msgr2=0x7efe040fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:13.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.421+0000 7efe0c000700 1 -- 172.21.15.27:0/2913786841 shutdown_connections 2024-04-16T05:43:13.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.421+0000 7efe0c000700 1 -- 172.21.15.27:0/2913786841 wait complete. 2024-04-16T05:43:13.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.422+0000 7efe0c000700 1 Processor -- start 2024-04-16T05:43:13.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.422+0000 7efe0c000700 1 -- start start 2024-04-16T05:43:13.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.423+0000 7efe0c000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe040fce40 0x7efe0410fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:13.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.423+0000 7efe0c000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe040fe040 0x7efe04110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:13.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.423+0000 7efe0c000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe04110850 con 0x7efe040fce40 2024-04-16T05:43:13.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.423+0000 7efe0c000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe04110f90 con 0x7efe040fe040 2024-04-16T05:43:13.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.423+0000 7efe09600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe040fce40 0x7efe0410fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:13.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.423+0000 7efe09600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe040fce40 0x7efe0410fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57972/0 (socket says 172.21.15.27:57972) 2024-04-16T05:43:13.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.423+0000 7efe09600700 1 -- 172.21.15.27:0/1003676356 learned_addr learned my addr 172.21.15.27:0/1003676356 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:43:13.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.424+0000 7efe08c00700 1 --2- 172.21.15.27:0/1003676356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe040fe040 0x7efe04110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:13.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.424+0000 7efe09600700 1 -- 172.21.15.27:0/1003676356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe040fe040 msgr2=0x7efe04110250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:13.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.424+0000 7efe09600700 1 --2- 172.21.15.27:0/1003676356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe040fe040 0x7efe04110250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:13.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.424+0000 7efe09600700 1 -- 172.21.15.27:0/1003676356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf8005040 con 0x7efe040fce40 2024-04-16T05:43:13.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.424+0000 7efe09600700 1 --2- 172.21.15.27:0/1003676356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe040fce40 0x7efe0410fd10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7efdf800b970 tx=0x7efdf8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:13.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.424+0000 7efe01600700 1 -- 172.21.15.27:0/1003676356 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdf800e040 con 0x7efe040fce40 2024-04-16T05:43:13.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.424+0000 7efe01600700 1 -- 172.21.15.27:0/1003676356 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efdf8004290 con 0x7efe040fce40 2024-04-16T05:43:13.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.424+0000 7efe0c000700 1 -- 172.21.15.27:0/1003676356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe04111210 con 0x7efe040fce40 2024-04-16T05:43:13.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.425+0000 7efe01600700 1 -- 172.21.15.27:0/1003676356 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdf801c630 con 0x7efe040fce40 2024-04-16T05:43:13.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.425+0000 7efe0c000700 1 -- 172.21.15.27:0/1003676356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe041117d0 con 0x7efe040fce40 2024-04-16T05:43:13.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.426+0000 7efe0c000700 1 -- 172.21.15.27:0/1003676356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe041019d0 con 0x7efe040fce40 2024-04-16T05:43:13.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.428+0000 7efe01600700 1 -- 172.21.15.27:0/1003676356 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 89652+0+0 (secure 0 0 0) 0x7efdf800bbf0 con 0x7efe040fce40 2024-04-16T05:43:13.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.429+0000 7efe01600700 1 --2- 172.21.15.27:0/1003676356 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7efdec06bcf0 0x7efdec06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:13.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.429+0000 7efe01600700 1 -- 172.21.15.27:0/1003676356 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7efdf808be40 con 0x7efe040fce40 2024-04-16T05:43:13.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.430+0000 7efe08c00700 1 --2- 172.21.15.27:0/1003676356 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7efdec06bcf0 0x7efdec06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:13.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.431+0000 7efe08c00700 1 --2- 172.21.15.27:0/1003676356 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7efdec06bcf0 0x7efdec06e1a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efdf0004990 tx=0x7efdf0009ee0 comp rx=0 tx=0).ready entity=mgr.24369 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:13.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.436+0000 7efe01600700 1 -- 172.21.15.27:0/1003676356 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7efdf8058f30 con 0x7efe040fce40 2024-04-16T05:43:13.751 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:43:13.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.751+0000 7efe0c000700 1 -- 172.21.15.27:0/1003676356 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7efe0405f5f0 con 0x7efe040fce40 2024-04-16T05:43:13.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.751+0000 7efe01600700 1 -- 172.21.15.27:0/1003676356 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7efdf8013e60 con 0x7efe040fce40 2024-04-16T05:43:13.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.754+0000 7efe0c000700 1 -- 172.21.15.27:0/1003676356 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7efdec06bcf0 msgr2=0x7efdec06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:13.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.754+0000 7efe0c000700 1 --2- 172.21.15.27:0/1003676356 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7efdec06bcf0 0x7efdec06e1a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efdf0004990 tx=0x7efdf0009ee0 comp rx=0 tx=0).stop 2024-04-16T05:43:13.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.754+0000 7efe0c000700 1 -- 172.21.15.27:0/1003676356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe040fce40 msgr2=0x7efe0410fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:13.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.754+0000 7efe0c000700 1 --2- 172.21.15.27:0/1003676356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe040fce40 0x7efe0410fd10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7efdf800b970 tx=0x7efdf8009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:13.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.754+0000 7efe0c000700 1 -- 172.21.15.27:0/1003676356 shutdown_connections 2024-04-16T05:43:13.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.754+0000 7efe0c000700 1 --2- 172.21.15.27:0/1003676356 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7efdec06bcf0 0x7efdec06e1a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:13.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.754+0000 7efe0c000700 1 --2- 172.21.15.27:0/1003676356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe040fce40 0x7efe0410fd10 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:13.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.754+0000 7efe0c000700 1 --2- 172.21.15.27:0/1003676356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efe040fe040 0x7efe04110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:13.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.754+0000 7efe0c000700 1 -- 172.21.15.27:0/1003676356 >> 172.21.15.27:0/1003676356 conn(0x7efe040f8530 msgr2=0x7efe040f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:13.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.755+0000 7efe0c000700 1 -- 172.21.15.27:0/1003676356 shutdown_connections 2024-04-16T05:43:13.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:13.755+0000 7efe0c000700 1 -- 172.21.15.27:0/1003676356 wait complete. 2024-04-16T05:43:14.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:14 smithi122 ceph-mon[35998]: pgmap v9: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-16T05:43:14.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:14 smithi122 ceph-mon[35998]: from='client.14680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:43:14.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:14 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:14.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:14 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:14.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:14 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/1003676356' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:43:14.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:14 smithi027 ceph-mon[29206]: pgmap v9: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-16T05:43:14.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:14 smithi027 ceph-mon[29206]: from='client.14680 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:43:14.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:14 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:14.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:14 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:14.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:14 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1003676356' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:43:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:16 smithi122 ceph-mon[35998]: pgmap v10: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T05:43:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:16 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:16 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:16 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:43:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:16 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:43:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:16 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:43:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:16 smithi122 ceph-mon[35998]: Adjusting osd_memory_target on smithi027 to 1557M 2024-04-16T05:43:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:16 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:16 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:43:16.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:16 smithi027 ceph-mon[29206]: pgmap v10: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T05:43:16.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:16 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:16.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:16 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:16.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:16 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:43:16.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:16 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:43:16.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:16 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:43:16.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:16 smithi027 ceph-mon[29206]: Adjusting osd_memory_target on smithi027 to 1557M 2024-04-16T05:43:16.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:16 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:16.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:16 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:43:18.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:18 smithi122 ceph-mon[35998]: pgmap v11: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T05:43:18.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:18 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:18.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:18 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:18.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:18 smithi027 ceph-mon[29206]: pgmap v11: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T05:43:18.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:18 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:18.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:18 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:20.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:20 smithi122 ceph-mon[35998]: pgmap v12: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:20.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:20 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:20.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:20 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:20.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:20 smithi027 ceph-mon[29206]: pgmap v12: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:20.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:20 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:20.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:20 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:22.753 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:22 smithi122 ceph-mon[35998]: pgmap v13: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:22.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:22 smithi027 ceph-mon[29206]: pgmap v13: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:23.864 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 68, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 66499 2024-04-16T05:43:23.864 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 68, success, outsize: 120 2024-04-16T05:43:23.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 70, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 66582 2024-04-16T05:43:23.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 70, success, outsize: 120 2024-04-16T05:43:23.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 72, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 66668 2024-04-16T05:43:23.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 72, success, outsize: 120 2024-04-16T05:43:23.927 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 74, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 66760 2024-04-16T05:43:23.927 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 74, success, outsize: 120 2024-04-16T05:43:23.950 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 76, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 66849 2024-04-16T05:43:23.950 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 76, success, outsize: 120 2024-04-16T05:43:23.966 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 78, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 66932 2024-04-16T05:43:23.967 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 78, success, outsize: 120 2024-04-16T05:43:23.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 80, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67018 2024-04-16T05:43:23.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 80, success, outsize: 120 2024-04-16T05:43:24.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 82, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67110 2024-04-16T05:43:24.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 82, success, outsize: 120 2024-04-16T05:43:24.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 84, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67199 2024-04-16T05:43:24.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 84, success, outsize: 120 2024-04-16T05:43:24.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 86, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67282 2024-04-16T05:43:24.047 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 86, success, outsize: 120 2024-04-16T05:43:24.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 88, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67368 2024-04-16T05:43:24.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 88, success, outsize: 120 2024-04-16T05:43:24.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 90, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67460 2024-04-16T05:43:24.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 90, success, outsize: 120 2024-04-16T05:43:24.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 92, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67549 2024-04-16T05:43:24.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 92, success, outsize: 120 2024-04-16T05:43:24.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 94, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67632 2024-04-16T05:43:24.130 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 94, success, outsize: 120 2024-04-16T05:43:24.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 96, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67718 2024-04-16T05:43:24.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 96, success, outsize: 120 2024-04-16T05:43:24.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 98, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67810 2024-04-16T05:43:24.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 98, success, outsize: 120 2024-04-16T05:43:24.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 100, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67899 2024-04-16T05:43:24.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 100, success, outsize: 120 2024-04-16T05:43:24.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 102, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 67982 2024-04-16T05:43:24.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 102, success, outsize: 120 2024-04-16T05:43:24.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 104, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68068 2024-04-16T05:43:24.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 104, success, outsize: 120 2024-04-16T05:43:24.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 106, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68160 2024-04-16T05:43:24.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 106, success, outsize: 120 2024-04-16T05:43:24.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 108, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68249 2024-04-16T05:43:24.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 108, success, outsize: 120 2024-04-16T05:43:24.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 110, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68332 2024-04-16T05:43:24.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 110, success, outsize: 120 2024-04-16T05:43:24.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 112, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68418 2024-04-16T05:43:24.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 112, success, outsize: 120 2024-04-16T05:43:24.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 114, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68510 2024-04-16T05:43:24.315 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 114, success, outsize: 120 2024-04-16T05:43:24.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 116, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68599 2024-04-16T05:43:24.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 116, success, outsize: 120 2024-04-16T05:43:24.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 118, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68682 2024-04-16T05:43:24.360 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 118, success, outsize: 120 2024-04-16T05:43:24.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 120, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68768 2024-04-16T05:43:24.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 120, success, outsize: 120 2024-04-16T05:43:24.402 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 122, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 68860 2024-04-16T05:43:24.403 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 122, success, outsize: 120 2024-04-16T05:43:24.746 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:24 smithi122 ceph-mon[35998]: pgmap v14: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:24.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:24 smithi027 ceph-mon[29206]: pgmap v14: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:26.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:26 smithi027 ceph-mon[29206]: pgmap v15: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:26.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:26 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:26.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:26 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:26.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:26 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:43:26.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:26 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:43:26.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:26 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:43:26.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:26 smithi027 ceph-mon[29206]: Adjusting osd_memory_target on smithi122 to 2581M 2024-04-16T05:43:26.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:26 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:26.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:26 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:43:26.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:26 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:43:26.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:26 smithi027 ceph-mon[29206]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-16T05:43:26.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:26 smithi027 ceph-mon[29206]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-16T05:43:26.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:26 smithi122 ceph-mon[35998]: pgmap v15: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:26.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:26 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:26.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:26 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:26.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:26 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:43:26.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:26 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:43:26.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:26 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-16T05:43:26.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:26 smithi122 ceph-mon[35998]: Adjusting osd_memory_target on smithi122 to 2581M 2024-04-16T05:43:26.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:26 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:26.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:26 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:43:26.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:26 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:43:26.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:26 smithi122 ceph-mon[35998]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-16T05:43:26.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:26 smithi122 ceph-mon[35998]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-16T05:43:27.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:27 smithi122 ceph-mon[35998]: Updating smithi122:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:43:27.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:27 smithi122 ceph-mon[35998]: Updating smithi027:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:43:27.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:27 smithi027 ceph-mon[29206]: Updating smithi122:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:43:27.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:27 smithi027 ceph-mon[29206]: Updating smithi027:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:43:28.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:28 smithi122 ceph-mon[35998]: pgmap v16: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:28.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:28 smithi122 ceph-mon[35998]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-16T05:43:28.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:28 smithi122 ceph-mon[35998]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-16T05:43:28.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:28 smithi027 ceph-mon[29206]: pgmap v16: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:28.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:28 smithi027 ceph-mon[29206]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-16T05:43:28.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:28 smithi027 ceph-mon[29206]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-16T05:43:29.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:29 smithi122 ceph-mon[35998]: Updating smithi122:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T05:43:29.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:29 smithi122 ceph-mon[35998]: Updating smithi027:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T05:43:29.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:29 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:29.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:29 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:29.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:29 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:29.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:29 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:29.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:29 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:29.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:29 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:29.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:29 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:29.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:29 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:29.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:29 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:29.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:29 smithi027 ceph-mon[29206]: Updating smithi122:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T05:43:29.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:29 smithi027 ceph-mon[29206]: Updating smithi027:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T05:43:29.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:29 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:29.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:29 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:29.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:29 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:29.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:29 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:29.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:29 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:29.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:29 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:29.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:29 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:29.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:29 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:29.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:29 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:30.701 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:30 smithi027 ceph-mon[29206]: pgmap v17: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:30.701 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:30 smithi027 ceph-mon[29206]: Reconfiguring node-exporter.smithi027 (dependencies changed)... 2024-04-16T05:43:30.702 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:30 smithi027 ceph-mon[29206]: Reconfiguring daemon node-exporter.smithi027 on smithi027 2024-04-16T05:43:30.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:30 smithi122 ceph-mon[35998]: pgmap v17: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:30.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:30 smithi122 ceph-mon[35998]: Reconfiguring node-exporter.smithi027 (dependencies changed)... 2024-04-16T05:43:30.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:30 smithi122 ceph-mon[35998]: Reconfiguring daemon node-exporter.smithi027 on smithi027 2024-04-16T05:43:31.733 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:31 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:43:31.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:31 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:43:32.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:32 smithi027 ceph-mon[29206]: pgmap v18: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:32.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:32 smithi122 ceph-mon[35998]: pgmap v18: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:34.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:34 smithi122 ceph-mon[35998]: pgmap v19: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:34.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:34 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:34.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:34 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:34.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:34 smithi122 ceph-mon[35998]: Reconfiguring alertmanager.smithi027 (dependencies changed)... 2024-04-16T05:43:34.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:34 smithi122 ceph-mon[35998]: Reconfiguring daemon alertmanager.smithi027 on smithi027 2024-04-16T05:43:34.529 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:34 smithi027 ceph-mon[29206]: pgmap v19: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:34.529 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:34 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:34.529 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:34 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:34.529 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:34 smithi027 ceph-mon[29206]: Reconfiguring alertmanager.smithi027 (dependencies changed)... 2024-04-16T05:43:34.530 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:34 smithi027 ceph-mon[29206]: Reconfiguring daemon alertmanager.smithi027 on smithi027 2024-04-16T05:43:36.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:36 smithi027 ceph-mon[29206]: pgmap v20: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:36.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:36 smithi122 ceph-mon[35998]: pgmap v20: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:38.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:38 smithi122 ceph-mon[35998]: pgmap v21: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:38.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:38 smithi027 ceph-mon[29206]: pgmap v21: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:40.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:40 smithi122 ceph-mon[35998]: pgmap v22: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:40.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:40 smithi027 ceph-mon[29206]: pgmap v22: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:42.450 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:42 smithi027 ceph-mon[29206]: pgmap v23: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:42.451 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:42 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:42.451 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:42 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:42.451 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:42 smithi027 ceph-mon[29206]: Reconfiguring grafana.smithi027 (dependencies changed)... 2024-04-16T05:43:42.451 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:42 smithi027 ceph-mon[29206]: Reconfiguring daemon grafana.smithi027 on smithi027 2024-04-16T05:43:42.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:42 smithi122 ceph-mon[35998]: pgmap v23: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:42.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:42 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:42.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:42 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:42.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:42 smithi122 ceph-mon[35998]: Reconfiguring grafana.smithi027 (dependencies changed)... 2024-04-16T05:43:42.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:42 smithi122 ceph-mon[35998]: Reconfiguring daemon grafana.smithi027 on smithi027 2024-04-16T05:43:43.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.926+0000 7f6f23e00700 1 -- 172.21.15.27:0/4216651625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f24100070 msgr2=0x7f6f241004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:43.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.926+0000 7f6f23e00700 1 --2- 172.21.15.27:0/4216651625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f24100070 0x7f6f241004e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f6f10004940 tx=0x7f6f10009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:43.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.926+0000 7f6f23e00700 1 -- 172.21.15.27:0/4216651625 shutdown_connections 2024-04-16T05:43:43.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.926+0000 7f6f23e00700 1 --2- 172.21.15.27:0/4216651625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f24100070 0x7f6f241004e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:43.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.926+0000 7f6f23e00700 1 --2- 172.21.15.27:0/4216651625 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f240fef00 0x7f6f240ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:43.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.926+0000 7f6f23e00700 1 -- 172.21.15.27:0/4216651625 >> 172.21.15.27:0/4216651625 conn(0x7f6f240fa790 msgr2=0x7f6f240fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:43.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.926+0000 7f6f23e00700 1 -- 172.21.15.27:0/4216651625 shutdown_connections 2024-04-16T05:43:43.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.926+0000 7f6f23e00700 1 -- 172.21.15.27:0/4216651625 wait complete. 2024-04-16T05:43:43.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.927+0000 7f6f23e00700 1 Processor -- start 2024-04-16T05:43:43.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.927+0000 7f6f23e00700 1 -- start start 2024-04-16T05:43:43.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.928+0000 7f6f23e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f240fef00 0x7f6f2410fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:43.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.928+0000 7f6f23e00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f24100070 0x7f6f2410ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:43.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.928+0000 7f6f23e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f241105b0 con 0x7f6f240fef00 2024-04-16T05:43:43.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.928+0000 7f6f23e00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f24110e00 con 0x7f6f24100070 2024-04-16T05:43:43.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.928+0000 7f6f22a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f240fef00 0x7f6f2410fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:43.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.928+0000 7f6f22a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f240fef00 0x7f6f2410fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39370/0 (socket says 172.21.15.27:39370) 2024-04-16T05:43:43.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.928+0000 7f6f22a00700 1 -- 172.21.15.27:0/1438629666 learned_addr learned my addr 172.21.15.27:0/1438629666 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:43:43.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.928+0000 7f6f22000700 1 --2- 172.21.15.27:0/1438629666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f24100070 0x7f6f2410ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:43.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.928+0000 7f6f22a00700 1 -- 172.21.15.27:0/1438629666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f24100070 msgr2=0x7f6f2410ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:43.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.929+0000 7f6f22a00700 1 --2- 172.21.15.27:0/1438629666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f24100070 0x7f6f2410ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:43.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.929+0000 7f6f22a00700 1 -- 172.21.15.27:0/1438629666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f10005040 con 0x7f6f240fef00 2024-04-16T05:43:43.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.929+0000 7f6f22a00700 1 --2- 172.21.15.27:0/1438629666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f240fef00 0x7f6f2410fa70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6f14002a50 tx=0x7f6f140046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:43.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.929+0000 7f6f1aa00700 1 -- 172.21.15.27:0/1438629666 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f14007cc0 con 0x7f6f240fef00 2024-04-16T05:43:43.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.929+0000 7f6f23e00700 1 -- 172.21.15.27:0/1438629666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f241110e0 con 0x7f6f240fef00 2024-04-16T05:43:43.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.929+0000 7f6f1aa00700 1 -- 172.21.15.27:0/1438629666 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f1400f040 con 0x7f6f240fef00 2024-04-16T05:43:43.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.929+0000 7f6f1aa00700 1 -- 172.21.15.27:0/1438629666 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f14009640 con 0x7f6f240fef00 2024-04-16T05:43:43.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.929+0000 7f6f23e00700 1 -- 172.21.15.27:0/1438629666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f24111620 con 0x7f6f240fef00 2024-04-16T05:43:43.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.930+0000 7f6f23e00700 1 -- 172.21.15.27:0/1438629666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f240fe290 con 0x7f6f240fef00 2024-04-16T05:43:43.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.932+0000 7f6f1aa00700 1 -- 172.21.15.27:0/1438629666 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 89652+0+0 (secure 0 0 0) 0x7f6f140097a0 con 0x7f6f240fef00 2024-04-16T05:43:43.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.933+0000 7f6f1aa00700 1 --2- 172.21.15.27:0/1438629666 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f6f0806bd00 0x7f6f0806e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:43.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.933+0000 7f6f1aa00700 1 -- 172.21.15.27:0/1438629666 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f6f1408a990 con 0x7f6f240fef00 2024-04-16T05:43:43.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.933+0000 7f6f22000700 1 --2- 172.21.15.27:0/1438629666 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f6f0806bd00 0x7f6f0806e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:43.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.935+0000 7f6f22000700 1 --2- 172.21.15.27:0/1438629666 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f6f0806bd00 0x7f6f0806e1b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6f10004f90 tx=0x7f6f10009e30 comp rx=0 tx=0).ready entity=mgr.24369 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:43.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:43.938+0000 7f6f1aa00700 1 -- 172.21.15.27:0/1438629666 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f6f14057980 con 0x7f6f240fef00 2024-04-16T05:43:44.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.198+0000 7f6f23e00700 1 -- 172.21.15.27:0/1438629666 --> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f241119c0 con 0x7f6f0806bd00 2024-04-16T05:43:44.198 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:44 smithi027 ceph-mon[29206]: pgmap v24: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:44.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.201+0000 7f6f1aa00700 1 -- 172.21.15.27:0/1438629666 <== mgr.24369 v2:172.21.15.122:6828/871482489 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7f6f241119c0 con 0x7f6f0806bd00 2024-04-16T05:43:44.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.204+0000 7f6f23e00700 1 -- 172.21.15.27:0/1438629666 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f6f0806bd00 msgr2=0x7f6f0806e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:44.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.204+0000 7f6f23e00700 1 --2- 172.21.15.27:0/1438629666 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f6f0806bd00 0x7f6f0806e1b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6f10004f90 tx=0x7f6f10009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:44.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.204+0000 7f6f23e00700 1 -- 172.21.15.27:0/1438629666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f240fef00 msgr2=0x7f6f2410fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:44.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.204+0000 7f6f23e00700 1 --2- 172.21.15.27:0/1438629666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f240fef00 0x7f6f2410fa70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6f14002a50 tx=0x7f6f140046b0 comp rx=0 tx=0).stop 2024-04-16T05:43:44.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.204+0000 7f6f23e00700 1 -- 172.21.15.27:0/1438629666 shutdown_connections 2024-04-16T05:43:44.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.204+0000 7f6f23e00700 1 --2- 172.21.15.27:0/1438629666 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f6f0806bd00 0x7f6f0806e1b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:44.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.204+0000 7f6f23e00700 1 --2- 172.21.15.27:0/1438629666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f240fef00 0x7f6f2410fa70 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:44.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.204+0000 7f6f23e00700 1 --2- 172.21.15.27:0/1438629666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f24100070 0x7f6f2410ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:44.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.204+0000 7f6f23e00700 1 -- 172.21.15.27:0/1438629666 >> 172.21.15.27:0/1438629666 conn(0x7f6f240fa790 msgr2=0x7f6f240fce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:44.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.204+0000 7f6f23e00700 1 -- 172.21.15.27:0/1438629666 shutdown_connections 2024-04-16T05:43:44.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.204+0000 7f6f23e00700 1 -- 172.21.15.27:0/1438629666 wait complete. 2024-04-16T05:43:44.220 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:43:44.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.361+0000 7fa133200700 1 -- 172.21.15.27:0/2389296017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1340fef00 msgr2=0x7fa1340ff310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:44.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.361+0000 7fa133200700 1 --2- 172.21.15.27:0/2389296017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1340fef00 0x7fa1340ff310 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa124004940 tx=0x7fa124009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:44.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.361+0000 7fa133200700 1 -- 172.21.15.27:0/2389296017 shutdown_connections 2024-04-16T05:43:44.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.361+0000 7fa133200700 1 --2- 172.21.15.27:0/2389296017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa134100070 0x7fa1341004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:44.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.361+0000 7fa133200700 1 --2- 172.21.15.27:0/2389296017 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1340fef00 0x7fa1340ff310 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:44.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.361+0000 7fa133200700 1 -- 172.21.15.27:0/2389296017 >> 172.21.15.27:0/2389296017 conn(0x7fa1340fa790 msgr2=0x7fa1340fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:44.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.361+0000 7fa133200700 1 -- 172.21.15.27:0/2389296017 shutdown_connections 2024-04-16T05:43:44.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.362+0000 7fa133200700 1 -- 172.21.15.27:0/2389296017 wait complete. 2024-04-16T05:43:44.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.362+0000 7fa133200700 1 Processor -- start 2024-04-16T05:43:44.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.363+0000 7fa133200700 1 -- start start 2024-04-16T05:43:44.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.363+0000 7fa133200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1340fef00 0x7fa13410fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:44.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.363+0000 7fa133200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa134100070 0x7fa13410ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:44.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.363+0000 7fa133200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1341105a0 con 0x7fa1340fef00 2024-04-16T05:43:44.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.363+0000 7fa133200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa134110df0 con 0x7fa134100070 2024-04-16T05:43:44.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.363+0000 7fa131e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1340fef00 0x7fa13410fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:44.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.363+0000 7fa131e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1340fef00 0x7fa13410fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60644/0 (socket says 172.21.15.27:60644) 2024-04-16T05:43:44.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.363+0000 7fa131e00700 1 -- 172.21.15.27:0/3029057375 learned_addr learned my addr 172.21.15.27:0/3029057375 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:43:44.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.364+0000 7fa131400700 1 --2- 172.21.15.27:0/3029057375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa134100070 0x7fa13410ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:44.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.364+0000 7fa131e00700 1 -- 172.21.15.27:0/3029057375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa134100070 msgr2=0x7fa13410ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:44.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.364+0000 7fa131e00700 1 --2- 172.21.15.27:0/3029057375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa134100070 0x7fa13410ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:44.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.364+0000 7fa131e00700 1 -- 172.21.15.27:0/3029057375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa124005040 con 0x7fa1340fef00 2024-04-16T05:43:44.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.364+0000 7fa131e00700 1 --2- 172.21.15.27:0/3029057375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1340fef00 0x7fa13410fa60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa12400b3b0 tx=0x7fa124009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:44.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.364+0000 7fa12a000700 1 -- 172.21.15.27:0/3029057375 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa12400e040 con 0x7fa1340fef00 2024-04-16T05:43:44.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.364+0000 7fa12a000700 1 -- 172.21.15.27:0/3029057375 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa124004290 con 0x7fa1340fef00 2024-04-16T05:43:44.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.364+0000 7fa133200700 1 -- 172.21.15.27:0/3029057375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa134111070 con 0x7fa1340fef00 2024-04-16T05:43:44.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.364+0000 7fa12a000700 1 -- 172.21.15.27:0/3029057375 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa12401c630 con 0x7fa1340fef00 2024-04-16T05:43:44.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.365+0000 7fa133200700 1 -- 172.21.15.27:0/3029057375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1341115b0 con 0x7fa1340fef00 2024-04-16T05:43:44.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.366+0000 7fa133200700 1 -- 172.21.15.27:0/3029057375 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1340fe290 con 0x7fa1340fef00 2024-04-16T05:43:44.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.367+0000 7fa12a000700 1 -- 172.21.15.27:0/3029057375 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 89652+0+0 (secure 0 0 0) 0x7fa12400bbf0 con 0x7fa1340fef00 2024-04-16T05:43:44.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.368+0000 7fa12a000700 1 --2- 172.21.15.27:0/3029057375 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fa11806bd00 0x7fa11806e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:44.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.368+0000 7fa12a000700 1 -- 172.21.15.27:0/3029057375 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fa12408bf30 con 0x7fa1340fef00 2024-04-16T05:43:44.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.368+0000 7fa131400700 1 --2- 172.21.15.27:0/3029057375 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fa11806bd00 0x7fa11806e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:44.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.370+0000 7fa131400700 1 --2- 172.21.15.27:0/3029057375 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fa11806bd00 0x7fa11806e1b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa120004990 tx=0x7fa120009ee0 comp rx=0 tx=0).ready entity=mgr.24369 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:44.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.373+0000 7fa12a000700 1 -- 172.21.15.27:0/3029057375 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fa124059020 con 0x7fa1340fef00 2024-04-16T05:43:44.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:44 smithi122 ceph-mon[35998]: pgmap v24: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:44.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.626+0000 7fa133200700 1 -- 172.21.15.27:0/3029057375 --> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa134111950 con 0x7fa11806bd00 2024-04-16T05:43:44.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.630+0000 7fa12a000700 1 -- 172.21.15.27:0/3029057375 <== mgr.24369 v2:172.21.15.122:6828/871482489 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7fa134111950 con 0x7fa11806bd00 2024-04-16T05:43:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.632+0000 7fa133200700 1 -- 172.21.15.27:0/3029057375 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fa11806bd00 msgr2=0x7fa11806e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.632+0000 7fa133200700 1 --2- 172.21.15.27:0/3029057375 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fa11806bd00 0x7fa11806e1b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa120004990 tx=0x7fa120009ee0 comp rx=0 tx=0).stop 2024-04-16T05:43:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.632+0000 7fa133200700 1 -- 172.21.15.27:0/3029057375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1340fef00 msgr2=0x7fa13410fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.632+0000 7fa133200700 1 --2- 172.21.15.27:0/3029057375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1340fef00 0x7fa13410fa60 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa12400b3b0 tx=0x7fa124009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.632+0000 7fa133200700 1 -- 172.21.15.27:0/3029057375 shutdown_connections 2024-04-16T05:43:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.632+0000 7fa133200700 1 --2- 172.21.15.27:0/3029057375 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fa11806bd00 0x7fa11806e1b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.632+0000 7fa133200700 1 --2- 172.21.15.27:0/3029057375 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1340fef00 0x7fa13410fa60 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.632+0000 7fa133200700 1 --2- 172.21.15.27:0/3029057375 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa134100070 0x7fa13410ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.632+0000 7fa133200700 1 -- 172.21.15.27:0/3029057375 >> 172.21.15.27:0/3029057375 conn(0x7fa1340fa790 msgr2=0x7fa1340fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.633+0000 7fa133200700 1 -- 172.21.15.27:0/3029057375 shutdown_connections 2024-04-16T05:43:44.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.633+0000 7fa133200700 1 -- 172.21.15.27:0/3029057375 wait complete. 2024-04-16T05:43:44.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.786+0000 7f998d400700 1 -- 172.21.15.27:0/2901015461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9988070870 msgr2=0x7f9988070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:44.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.786+0000 7f998d400700 1 --2- 172.21.15.27:0/2901015461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9988070870 0x7f9988070ce0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9970004940 tx=0x7f9970009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:44.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.786+0000 7f998d400700 1 -- 172.21.15.27:0/2901015461 shutdown_connections 2024-04-16T05:43:44.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.786+0000 7f998d400700 1 --2- 172.21.15.27:0/2901015461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9988070870 0x7f9988070ce0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:44.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.786+0000 7f998d400700 1 --2- 172.21.15.27:0/2901015461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9988071e30 0x7f9988070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:44.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.786+0000 7f998d400700 1 -- 172.21.15.27:0/2901015461 >> 172.21.15.27:0/2901015461 conn(0x7f99880f8540 msgr2=0x7f99880fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:44.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.786+0000 7f998d400700 1 -- 172.21.15.27:0/2901015461 shutdown_connections 2024-04-16T05:43:44.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.786+0000 7f998d400700 1 -- 172.21.15.27:0/2901015461 wait complete. 2024-04-16T05:43:44.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.787+0000 7f998d400700 1 Processor -- start 2024-04-16T05:43:44.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.787+0000 7f998d400700 1 -- start start 2024-04-16T05:43:44.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.788+0000 7f998d400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9988070870 0x7f998810fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:44.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.788+0000 7f998d400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9988071e30 0x7f99881102a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:44.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.788+0000 7f998d400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99881108a0 con 0x7f9988070870 2024-04-16T05:43:44.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.788+0000 7f998d400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9988110fe0 con 0x7f9988071e30 2024-04-16T05:43:44.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.788+0000 7f9986800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9988070870 0x7f998810fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:44.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.788+0000 7f9986800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9988070870 0x7f998810fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60672/0 (socket says 172.21.15.27:60672) 2024-04-16T05:43:44.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.788+0000 7f9986800700 1 -- 172.21.15.27:0/3367352966 learned_addr learned my addr 172.21.15.27:0/3367352966 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:43:44.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.788+0000 7f997fe00700 1 --2- 172.21.15.27:0/3367352966 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9988071e30 0x7f99881102a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:44.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.788+0000 7f9986800700 1 -- 172.21.15.27:0/3367352966 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9988071e30 msgr2=0x7f99881102a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:44.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.788+0000 7f9986800700 1 --2- 172.21.15.27:0/3367352966 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9988071e30 0x7f99881102a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:44.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.788+0000 7f9986800700 1 -- 172.21.15.27:0/3367352966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9970005040 con 0x7f9988070870 2024-04-16T05:43:44.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.788+0000 7f997fe00700 1 --2- 172.21.15.27:0/3367352966 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9988071e30 0x7f99881102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:43:44.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.789+0000 7f9986800700 1 --2- 172.21.15.27:0/3367352966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9988070870 0x7f998810fd60 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f997800c800 tx=0x7f9978004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:44.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.789+0000 7f997ea00700 1 -- 172.21.15.27:0/3367352966 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f997800a760 con 0x7f9988070870 2024-04-16T05:43:44.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.789+0000 7f997ea00700 1 -- 172.21.15.27:0/3367352966 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f997800ad90 con 0x7f9988070870 2024-04-16T05:43:44.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.789+0000 7f998d400700 1 -- 172.21.15.27:0/3367352966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99881112c0 con 0x7f9988070870 2024-04-16T05:43:44.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.789+0000 7f997ea00700 1 -- 172.21.15.27:0/3367352966 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99780113a0 con 0x7f9988070870 2024-04-16T05:43:44.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.789+0000 7f998d400700 1 -- 172.21.15.27:0/3367352966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9988111880 con 0x7f9988070870 2024-04-16T05:43:44.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.790+0000 7f998d400700 1 -- 172.21.15.27:0/3367352966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f998805f5f0 con 0x7f9988070870 2024-04-16T05:43:44.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.791+0000 7f997ea00700 1 -- 172.21.15.27:0/3367352966 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 89652+0+0 (secure 0 0 0) 0x7f9978012400 con 0x7f9988070870 2024-04-16T05:43:44.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.792+0000 7f997ea00700 1 --2- 172.21.15.27:0/3367352966 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f996c06bcf0 0x7f996c06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:44.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.793+0000 7f997ea00700 1 -- 172.21.15.27:0/3367352966 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f997808a8f0 con 0x7f9988070870 2024-04-16T05:43:44.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.793+0000 7f997fe00700 1 --2- 172.21.15.27:0/3367352966 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f996c06bcf0 0x7f996c06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:44.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.794+0000 7f997fe00700 1 --2- 172.21.15.27:0/3367352966 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f996c06bcf0 0x7f996c06e1a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9970000c00 tx=0x7f9970009e30 comp rx=0 tx=0).ready entity=mgr.24369 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:44.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:44.797+0000 7f997ea00700 1 -- 172.21.15.27:0/3367352966 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f9978057bb0 con 0x7f9988070870 2024-04-16T05:43:45.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.024+0000 7f998d400700 1 -- 172.21.15.27:0/3367352966 --> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9988111d10 con 0x7f996c06bcf0 2024-04-16T05:43:45.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.054+0000 7f997ea00700 1 -- 172.21.15.27:0/3367352966 <== mgr.24369 v2:172.21.15.122:6828/871482489 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7f9988111d10 con 0x7f996c06bcf0 2024-04-16T05:43:45.053 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:43:45.054 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 starting - - - - 2024-04-16T05:43:45.054 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (11m) 31s ago 11m 6665k - 17.2.7-847-g45317f95 b333581ccdf5 f1acb6959c63 2024-04-16T05:43:45.054 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (9m) 27s ago 9m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 e81efc20e8c0 2024-04-16T05:43:45.054 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (8m) 31s ago 10m 73.3M - 9.4.7 954c08fa6188 0de7ee73579d 2024-04-16T05:43:45.054 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (3m) 31s ago 3m 22.2M - 17.2.7-847-g45317f95 b333581ccdf5 0824e2621be5 2024-04-16T05:43:45.054 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (3m) 31s ago 3m 11.7M - 17.2.7-847-g45317f95 b333581ccdf5 b332a72cc594 2024-04-16T05:43:45.054 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (3m) 27s ago 3m 11.9M - 17.2.7-847-g45317f95 b333581ccdf5 f5e36815ddfa 2024-04-16T05:43:45.054 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (3m) 27s ago 3m 11.8M - 17.2.7-847-g45317f95 b333581ccdf5 755a73c65a7f 2024-04-16T05:43:45.054 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:9283 running (12m) 31s ago 12m 335M - 17.2.7-847-g45317f95 b333581ccdf5 78f461c198fe 2024-04-16T05:43:45.054 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283 running (55s) 27s ago 9m 488M - 19.0.0-2884-g218a6914 a4e4811815e4 b1a1b7a6d536 2024-04-16T05:43:45.054 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (12m) 31s ago 12m 47.1M 2048M 17.2.7-847-g45317f95 b333581ccdf5 88575dbb8d98 2024-04-16T05:43:45.054 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (9m) 27s ago 9m 35.9M 2048M 17.2.7-847-g45317f95 b333581ccdf5 5df19eab6d21 2024-04-16T05:43:45.054 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 starting - - - - 2024-04-16T05:43:45.054 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (9m) 27s ago 9m 7470k - 1.5.0 0da6a335fe13 7f796c66dff6 2024-04-16T05:43:45.055 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (8m) 31s ago 8m 46.4M 1557M 17.2.7-847-g45317f95 b333581ccdf5 15e049573bca 2024-04-16T05:43:45.055 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (7m) 31s ago 7m 46.1M 1557M 17.2.7-847-g45317f95 b333581ccdf5 e146295f6500 2024-04-16T05:43:45.055 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (7m) 31s ago 7m 47.5M 1557M 17.2.7-847-g45317f95 b333581ccdf5 6212f6a18fa4 2024-04-16T05:43:45.055 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (6m) 27s ago 6m 44.8M 2581M 17.2.7-847-g45317f95 b333581ccdf5 d6724ad05f9f 2024-04-16T05:43:45.055 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (5m) 27s ago 5m 43.0M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7a59908a3e10 2024-04-16T05:43:45.055 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (5m) 27s ago 5m 42.3M 2581M 17.2.7-847-g45317f95 b333581ccdf5 8e4b60163907 2024-04-16T05:43:45.055 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (8m) 31s ago 10m 49.1M - 2.43.0 a07b618ecd1d 719577e1340d 2024-04-16T05:43:45.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.056+0000 7f998d400700 1 -- 172.21.15.27:0/3367352966 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f996c06bcf0 msgr2=0x7f996c06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:45.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.056+0000 7f998d400700 1 --2- 172.21.15.27:0/3367352966 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f996c06bcf0 0x7f996c06e1a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9970000c00 tx=0x7f9970009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:45.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.056+0000 7f998d400700 1 -- 172.21.15.27:0/3367352966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9988070870 msgr2=0x7f998810fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:45.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.057+0000 7f998d400700 1 --2- 172.21.15.27:0/3367352966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9988070870 0x7f998810fd60 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f997800c800 tx=0x7f9978004470 comp rx=0 tx=0).stop 2024-04-16T05:43:45.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.057+0000 7f998d400700 1 -- 172.21.15.27:0/3367352966 shutdown_connections 2024-04-16T05:43:45.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.057+0000 7f998d400700 1 --2- 172.21.15.27:0/3367352966 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f996c06bcf0 0x7f996c06e1a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:45.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.057+0000 7f998d400700 1 --2- 172.21.15.27:0/3367352966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9988070870 0x7f998810fd60 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:45.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.057+0000 7f998d400700 1 --2- 172.21.15.27:0/3367352966 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9988071e30 0x7f99881102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:45.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.057+0000 7f998d400700 1 -- 172.21.15.27:0/3367352966 >> 172.21.15.27:0/3367352966 conn(0x7f99880f8540 msgr2=0x7f99880f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:45.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.057+0000 7f998d400700 1 -- 172.21.15.27:0/3367352966 shutdown_connections 2024-04-16T05:43:45.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.057+0000 7f998d400700 1 -- 172.21.15.27:0/3367352966 wait complete. 2024-04-16T05:43:45.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.211+0000 7f6c97e00700 1 -- 172.21.15.27:0/4073543792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c900ff000 msgr2=0x7f6c900ff410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:45.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.211+0000 7f6c97e00700 1 --2- 172.21.15.27:0/4073543792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c900ff000 0x7f6c900ff410 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6c84004940 tx=0x7f6c84009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:45.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.212+0000 7f6c97e00700 1 -- 172.21.15.27:0/4073543792 shutdown_connections 2024-04-16T05:43:45.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.212+0000 7f6c97e00700 1 --2- 172.21.15.27:0/4073543792 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c901002b0 0x7f6c90100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:45.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.212+0000 7f6c97e00700 1 --2- 172.21.15.27:0/4073543792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c900ff000 0x7f6c900ff410 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:45.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.212+0000 7f6c97e00700 1 -- 172.21.15.27:0/4073543792 >> 172.21.15.27:0/4073543792 conn(0x7f6c900fa950 msgr2=0x7f6c900fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:45.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.212+0000 7f6c97e00700 1 -- 172.21.15.27:0/4073543792 shutdown_connections 2024-04-16T05:43:45.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.212+0000 7f6c97e00700 1 -- 172.21.15.27:0/4073543792 wait complete. 2024-04-16T05:43:45.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.213+0000 7f6c97e00700 1 Processor -- start 2024-04-16T05:43:45.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.213+0000 7f6c97e00700 1 -- start start 2024-04-16T05:43:45.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.214+0000 7f6c97e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c900ff000 0x7f6c9010fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:45.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.214+0000 7f6c97e00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c901002b0 0x7f6c90110160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:45.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.214+0000 7f6c97e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c90110760 con 0x7f6c900ff000 2024-04-16T05:43:45.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.214+0000 7f6c97e00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c90110fb0 con 0x7f6c901002b0 2024-04-16T05:43:45.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.214+0000 7f6c95400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c900ff000 0x7f6c9010fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:45.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.214+0000 7f6c95400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c900ff000 0x7f6c9010fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60694/0 (socket says 172.21.15.27:60694) 2024-04-16T05:43:45.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.214+0000 7f6c95400700 1 -- 172.21.15.27:0/2497354809 learned_addr learned my addr 172.21.15.27:0/2497354809 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:43:45.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.214+0000 7f6c94a00700 1 --2- 172.21.15.27:0/2497354809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c901002b0 0x7f6c90110160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:45.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.215+0000 7f6c95400700 1 -- 172.21.15.27:0/2497354809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c901002b0 msgr2=0x7f6c90110160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:45.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.215+0000 7f6c95400700 1 --2- 172.21.15.27:0/2497354809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c901002b0 0x7f6c90110160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:45.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.215+0000 7f6c95400700 1 -- 172.21.15.27:0/2497354809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c84005040 con 0x7f6c900ff000 2024-04-16T05:43:45.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.215+0000 7f6c95400700 1 --2- 172.21.15.27:0/2497354809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c900ff000 0x7f6c9010fc20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009150 tx=0x7f6c84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:45.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.215+0000 7f6c8d600700 1 -- 172.21.15.27:0/2497354809 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c8400e040 con 0x7f6c900ff000 2024-04-16T05:43:45.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.215+0000 7f6c8d600700 1 -- 172.21.15.27:0/2497354809 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c8400bbf0 con 0x7f6c900ff000 2024-04-16T05:43:45.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.215+0000 7f6c8d600700 1 -- 172.21.15.27:0/2497354809 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c8401c630 con 0x7f6c900ff000 2024-04-16T05:43:45.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.215+0000 7f6c97e00700 1 -- 172.21.15.27:0/2497354809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c90111230 con 0x7f6c900ff000 2024-04-16T05:43:45.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.216+0000 7f6c97e00700 1 -- 172.21.15.27:0/2497354809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c90111770 con 0x7f6c900ff000 2024-04-16T05:43:45.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.217+0000 7f6c97e00700 1 -- 172.21.15.27:0/2497354809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c9005f5f0 con 0x7f6c900ff000 2024-04-16T05:43:45.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.218+0000 7f6c8d600700 1 -- 172.21.15.27:0/2497354809 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 89652+0+0 (secure 0 0 0) 0x7f6c8400bd60 con 0x7f6c900ff000 2024-04-16T05:43:45.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.220+0000 7f6c8d600700 1 --2- 172.21.15.27:0/2497354809 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f6c7806bcf0 0x7f6c7806e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:45.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.220+0000 7f6c8d600700 1 -- 172.21.15.27:0/2497354809 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f6c8408be30 con 0x7f6c900ff000 2024-04-16T05:43:45.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.220+0000 7f6c94a00700 1 --2- 172.21.15.27:0/2497354809 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f6c7806bcf0 0x7f6c7806e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:45.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.222+0000 7f6c94a00700 1 --2- 172.21.15.27:0/2497354809 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f6c7806bcf0 0x7f6c7806e1a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c007fd0 tx=0x7f6c7c009040 comp rx=0 tx=0).ready entity=mgr.24369 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:45.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.225+0000 7f6c8d600700 1 -- 172.21.15.27:0/2497354809 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f6c84059090 con 0x7f6c900ff000 2024-04-16T05:43:45.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:45 smithi027 ceph-mon[29206]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:43:45.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:45 smithi122 ceph-mon[35998]: from='client.14688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:43:45.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.582+0000 7f6c97e00700 1 -- 172.21.15.27:0/2497354809 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6c90107af0 con 0x7f6c900ff000 2024-04-16T05:43:45.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.583+0000 7f6c8d600700 1 -- 172.21.15.27:0/2497354809 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+846 (secure 0 0 0) 0x7f6c84058c20 con 0x7f6c900ff000 2024-04-16T05:43:45.582 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:43:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:43:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T05:43:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:43:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:43:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 1, 2024-04-16T05:43:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 1 2024-04-16T05:43:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:43:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:43:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T05:43:45.582 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:43:45.583 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:43:45.583 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T05:43:45.583 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:43:45.583 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:43:45.583 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 13, 2024-04-16T05:43:45.583 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 1 2024-04-16T05:43:45.583 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:43:45.583 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:43:45.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.585+0000 7f6c97e00700 1 -- 172.21.15.27:0/2497354809 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f6c7806bcf0 msgr2=0x7f6c7806e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:45.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.585+0000 7f6c97e00700 1 --2- 172.21.15.27:0/2497354809 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f6c7806bcf0 0x7f6c7806e1a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c007fd0 tx=0x7f6c7c009040 comp rx=0 tx=0).stop 2024-04-16T05:43:45.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.585+0000 7f6c97e00700 1 -- 172.21.15.27:0/2497354809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c900ff000 msgr2=0x7f6c9010fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:45.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.585+0000 7f6c97e00700 1 --2- 172.21.15.27:0/2497354809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c900ff000 0x7f6c9010fc20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009150 tx=0x7f6c84009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:45.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.585+0000 7f6c97e00700 1 -- 172.21.15.27:0/2497354809 shutdown_connections 2024-04-16T05:43:45.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.585+0000 7f6c97e00700 1 --2- 172.21.15.27:0/2497354809 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f6c7806bcf0 0x7f6c7806e1a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:45.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.585+0000 7f6c97e00700 1 --2- 172.21.15.27:0/2497354809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c900ff000 0x7f6c9010fc20 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:45.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.585+0000 7f6c97e00700 1 --2- 172.21.15.27:0/2497354809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c901002b0 0x7f6c90110160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:45.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.585+0000 7f6c97e00700 1 -- 172.21.15.27:0/2497354809 >> 172.21.15.27:0/2497354809 conn(0x7f6c900fa950 msgr2=0x7f6c900fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:45.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.586+0000 7f6c97e00700 1 -- 172.21.15.27:0/2497354809 shutdown_connections 2024-04-16T05:43:45.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.586+0000 7f6c97e00700 1 -- 172.21.15.27:0/2497354809 wait complete. 2024-04-16T05:43:45.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.740+0000 7f5941800700 1 -- 172.21.15.27:0/3787904247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f593c0fe0e0 msgr2=0x7f593c0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:45.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.740+0000 7f5941800700 1 --2- 172.21.15.27:0/3787904247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f593c0fe0e0 0x7f593c0fe550 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5928004940 tx=0x7f5928009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:45.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.740+0000 7f5941800700 1 -- 172.21.15.27:0/3787904247 shutdown_connections 2024-04-16T05:43:45.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.740+0000 7f5941800700 1 --2- 172.21.15.27:0/3787904247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f593c0fe0e0 0x7f593c0fe550 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:45.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.740+0000 7f5941800700 1 --2- 172.21.15.27:0/3787904247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f593c0fcef0 0x7f593c0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.740+0000 7f5941800700 1 -- 172.21.15.27:0/3787904247 >> 172.21.15.27:0/3787904247 conn(0x7f593c0f8530 msgr2=0x7f593c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.740+0000 7f5941800700 1 -- 172.21.15.27:0/3787904247 shutdown_connections 2024-04-16T05:43:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.740+0000 7f5941800700 1 -- 172.21.15.27:0/3787904247 wait complete. 2024-04-16T05:43:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.741+0000 7f5941800700 1 Processor -- start 2024-04-16T05:43:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.741+0000 7f5941800700 1 -- start start 2024-04-16T05:43:45.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.742+0000 7f5941800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f593c0fcef0 0x7f593c10da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:45.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.742+0000 7f5941800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f593c0fe0e0 0x7f593c10dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:45.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.742+0000 7f5941800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c10e5d0 con 0x7f593c0fe0e0 2024-04-16T05:43:45.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.742+0000 7f5941800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c10edd0 con 0x7f593c0fcef0 2024-04-16T05:43:45.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.742+0000 7f593a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f593c0fe0e0 0x7f593c10dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:45.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.742+0000 7f593a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f593c0fe0e0 0x7f593c10dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60704/0 (socket says 172.21.15.27:60704) 2024-04-16T05:43:45.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.742+0000 7f593a000700 1 -- 172.21.15.27:0/1474701039 learned_addr learned my addr 172.21.15.27:0/1474701039 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:43:45.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.742+0000 7f593aa00700 1 --2- 172.21.15.27:0/1474701039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f593c0fcef0 0x7f593c10da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:45.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.742+0000 7f593a000700 1 -- 172.21.15.27:0/1474701039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f593c0fcef0 msgr2=0x7f593c10da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:45.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.743+0000 7f593a000700 1 --2- 172.21.15.27:0/1474701039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f593c0fcef0 0x7f593c10da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:45.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.743+0000 7f593a000700 1 -- 172.21.15.27:0/1474701039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5928005040 con 0x7f593c0fe0e0 2024-04-16T05:43:45.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.743+0000 7f593a000700 1 --2- 172.21.15.27:0/1474701039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f593c0fe0e0 0x7f593c10dfd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f592800b3b0 tx=0x7f5928009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:45.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.743+0000 7f5932a00700 1 -- 172.21.15.27:0/1474701039 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f592800e040 con 0x7f593c0fe0e0 2024-04-16T05:43:45.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.743+0000 7f5932a00700 1 -- 172.21.15.27:0/1474701039 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5928004290 con 0x7f593c0fe0e0 2024-04-16T05:43:45.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.743+0000 7f5932a00700 1 -- 172.21.15.27:0/1474701039 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f592801c630 con 0x7f593c0fe0e0 2024-04-16T05:43:45.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.743+0000 7f5941800700 1 -- 172.21.15.27:0/1474701039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f593c10f050 con 0x7f593c0fe0e0 2024-04-16T05:43:45.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.744+0000 7f5941800700 1 -- 172.21.15.27:0/1474701039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f593c10f590 con 0x7f593c0fe0e0 2024-04-16T05:43:45.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.745+0000 7f5941800700 1 -- 172.21.15.27:0/1474701039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f593c05f5f0 con 0x7f593c0fe0e0 2024-04-16T05:43:45.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.746+0000 7f5932a00700 1 -- 172.21.15.27:0/1474701039 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 89652+0+0 (secure 0 0 0) 0x7f592800bbf0 con 0x7f593c0fe0e0 2024-04-16T05:43:45.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.747+0000 7f5932a00700 1 --2- 172.21.15.27:0/1474701039 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f592006bcf0 0x7f592006e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:45.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.747+0000 7f5932a00700 1 -- 172.21.15.27:0/1474701039 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f592808bba0 con 0x7f593c0fe0e0 2024-04-16T05:43:45.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.748+0000 7f593aa00700 1 --2- 172.21.15.27:0/1474701039 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f592006bcf0 0x7f592006e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:45.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.749+0000 7f593aa00700 1 --2- 172.21.15.27:0/1474701039 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f592006bcf0 0x7f592006e1a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f592c009fd0 tx=0x7f592c009f00 comp rx=0 tx=0).ready entity=mgr.24369 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:45.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:45.752+0000 7f5932a00700 1 -- 172.21.15.27:0/1474701039 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f5928058d50 con 0x7f593c0fe0e0 2024-04-16T05:43:46.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.032+0000 7f5941800700 1 -- 172.21.15.27:0/1474701039 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f593c10f8f0 con 0x7f593c0fe0e0 2024-04-16T05:43:46.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.033+0000 7f5932a00700 1 -- 172.21.15.27:0/1474701039 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1810 (secure 0 0 0) 0x7f59280588e0 con 0x7f593c0fe0e0 2024-04-16T05:43:46.033 INFO:teuthology.orchestra.run.smithi027.stdout:e16 2024-04-16T05:43:46.033 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:43:46.033 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:43:46.033 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:43:46.033 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:43:46.034 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:43:46.034 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:43:46.034 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 15 2024-04-16T05:43:46.034 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:43:46.034 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:43:46.034 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:40:32.701649+0000 2024-04-16T05:43:46.034 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:43:46.034 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:43:46.034 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:43:46.034 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:43:46.034 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:43:46.034 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:43:46.034 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:43:46.034 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 46 2024-04-16T05:43:46.035 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:43:46.035 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:43:46.035 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:43:46.035 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=14462} 2024-04-16T05:43:46.035 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:43:46.035 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:43:46.035 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:43:46.035 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:43:46.035 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:43:46.035 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:43:46.035 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:43:46.035 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:43:46.035 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:43:46.035 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:43:46.036 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:43:46.036 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:43:46.036 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:43:46.036 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{-1:14474} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2065690722,v1:172.21.15.27:6827/2065690722] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:43:46.036 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:24263} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:43:46.036 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:24273} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:43:46.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.036+0000 7f5941800700 1 -- 172.21.15.27:0/1474701039 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f592006bcf0 msgr2=0x7f592006e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:46.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.036+0000 7f5941800700 1 --2- 172.21.15.27:0/1474701039 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f592006bcf0 0x7f592006e1a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f592c009fd0 tx=0x7f592c009f00 comp rx=0 tx=0).stop 2024-04-16T05:43:46.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.036+0000 7f5941800700 1 -- 172.21.15.27:0/1474701039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f593c0fe0e0 msgr2=0x7f593c10dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:46.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.036+0000 7f5941800700 1 --2- 172.21.15.27:0/1474701039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f593c0fe0e0 0x7f593c10dfd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f592800b3b0 tx=0x7f5928009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:46.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.037+0000 7f5941800700 1 -- 172.21.15.27:0/1474701039 shutdown_connections 2024-04-16T05:43:46.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.037+0000 7f5941800700 1 --2- 172.21.15.27:0/1474701039 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f593c0fcef0 0x7f593c10da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:46.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.037+0000 7f5941800700 1 --2- 172.21.15.27:0/1474701039 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f592006bcf0 0x7f592006e1a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:46.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.037+0000 7f5941800700 1 --2- 172.21.15.27:0/1474701039 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f593c0fe0e0 0x7f593c10dfd0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:46.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.037+0000 7f5941800700 1 -- 172.21.15.27:0/1474701039 >> 172.21.15.27:0/1474701039 conn(0x7f593c0f8530 msgr2=0x7f593c0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:46.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.037+0000 7f5941800700 1 -- 172.21.15.27:0/1474701039 shutdown_connections 2024-04-16T05:43:46.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.037+0000 7f5941800700 1 -- 172.21.15.27:0/1474701039 wait complete. 2024-04-16T05:43:46.038 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2024-04-16T05:43:46.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.192+0000 7fd014400700 1 -- 172.21.15.27:0/2736932154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00c0fe040 msgr2=0x7fd00c0fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:46.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.192+0000 7fd014400700 1 --2- 172.21.15.27:0/2736932154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00c0fe040 0x7fd00c0fe4b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fcff8004940 tx=0x7fcff8009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:46.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.192+0000 7fd014400700 1 -- 172.21.15.27:0/2736932154 shutdown_connections 2024-04-16T05:43:46.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.192+0000 7fd014400700 1 --2- 172.21.15.27:0/2736932154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00c0fe040 0x7fd00c0fe4b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:46.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.192+0000 7fd014400700 1 --2- 172.21.15.27:0/2736932154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd00c0fce40 0x7fd00c0fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:46.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.192+0000 7fd014400700 1 -- 172.21.15.27:0/2736932154 >> 172.21.15.27:0/2736932154 conn(0x7fd00c0f8570 msgr2=0x7fd00c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:46.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.192+0000 7fd014400700 1 -- 172.21.15.27:0/2736932154 shutdown_connections 2024-04-16T05:43:46.192 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.192+0000 7fd014400700 1 -- 172.21.15.27:0/2736932154 wait complete. 2024-04-16T05:43:46.193 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.193+0000 7fd014400700 1 Processor -- start 2024-04-16T05:43:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.193+0000 7fd014400700 1 -- start start 2024-04-16T05:43:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.194+0000 7fd014400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00c0fce40 0x7fd00c10fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.194+0000 7fd014400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd00c0fe040 0x7fd00c110190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.194+0000 7fd014400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c110790 con 0x7fd00c0fce40 2024-04-16T05:43:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.194+0000 7fd014400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c110fe0 con 0x7fd00c0fe040 2024-04-16T05:43:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.194+0000 7fd011a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00c0fce40 0x7fd00c10fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.194+0000 7fd011a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00c0fce40 0x7fd00c10fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60724/0 (socket says 172.21.15.27:60724) 2024-04-16T05:43:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.194+0000 7fd011a00700 1 -- 172.21.15.27:0/1583678478 learned_addr learned my addr 172.21.15.27:0/1583678478 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:43:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.194+0000 7fd011000700 1 --2- 172.21.15.27:0/1583678478 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd00c0fe040 0x7fd00c110190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.194+0000 7fd011a00700 1 -- 172.21.15.27:0/1583678478 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd00c0fe040 msgr2=0x7fd00c110190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:46.194 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.194+0000 7fd011a00700 1 --2- 172.21.15.27:0/1583678478 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd00c0fe040 0x7fd00c110190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:46.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.194+0000 7fd011a00700 1 -- 172.21.15.27:0/1583678478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff8005040 con 0x7fd00c0fce40 2024-04-16T05:43:46.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.194+0000 7fd011000700 1 --2- 172.21.15.27:0/1583678478 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd00c0fe040 0x7fd00c110190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:43:46.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.195+0000 7fd011a00700 1 --2- 172.21.15.27:0/1583678478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00c0fce40 0x7fd00c10fc50 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd000002a80 tx=0x7fd000004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:46.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.195+0000 7fd009600700 1 -- 172.21.15.27:0/1583678478 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd000007cc0 con 0x7fd00c0fce40 2024-04-16T05:43:46.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.195+0000 7fd009600700 1 -- 172.21.15.27:0/1583678478 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd00000f040 con 0x7fd00c0fce40 2024-04-16T05:43:46.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.195+0000 7fd009600700 1 -- 172.21.15.27:0/1583678478 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd00000dad0 con 0x7fd00c0fce40 2024-04-16T05:43:46.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.195+0000 7fd014400700 1 -- 172.21.15.27:0/1583678478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd00c1112c0 con 0x7fd00c0fce40 2024-04-16T05:43:46.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.195+0000 7fd014400700 1 -- 172.21.15.27:0/1583678478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd00c111800 con 0x7fd00c0fce40 2024-04-16T05:43:46.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.196+0000 7fd014400700 1 -- 172.21.15.27:0/1583678478 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd00c1019d0 con 0x7fd00c0fce40 2024-04-16T05:43:46.196 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.197+0000 7fd009600700 1 -- 172.21.15.27:0/1583678478 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 89652+0+0 (secure 0 0 0) 0x7fd00000dc30 con 0x7fd00c0fce40 2024-04-16T05:43:46.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.198+0000 7fd009600700 1 --2- 172.21.15.27:0/1583678478 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fcff406bcf0 0x7fcff406e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:46.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.198+0000 7fd009600700 1 -- 172.21.15.27:0/1583678478 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fd00008a7c0 con 0x7fd00c0fce40 2024-04-16T05:43:46.198 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.199+0000 7fd011000700 1 --2- 172.21.15.27:0/1583678478 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fcff406bcf0 0x7fcff406e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:46.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.200+0000 7fd011000700 1 --2- 172.21.15.27:0/1583678478 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fcff406bcf0 0x7fcff406e1a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcff800b7a0 tx=0x7fcff8009e30 comp rx=0 tx=0).ready entity=mgr.24369 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:46.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.203+0000 7fd009600700 1 -- 172.21.15.27:0/1583678478 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fd0000578b0 con 0x7fd00c0fce40 2024-04-16T05:43:46.284 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:46 smithi027 ceph-mon[29206]: from='client.14692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:43:46.284 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:46 smithi027 ceph-mon[29206]: pgmap v25: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:46.284 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:46 smithi027 ceph-mon[29206]: from='client.14696 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:43:46.285 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:46 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/2497354809' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:43:46.285 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:46 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:43:46.285 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:46 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/1474701039' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:43:46.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.445+0000 7fd014400700 1 -- 172.21.15.27:0/1583678478 --> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd00c111a70 con 0x7fcff406bcf0 2024-04-16T05:43:46.448 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.449+0000 7fd009600700 1 -- 172.21.15.27:0/1583678478 <== mgr.24369 v2:172.21.15.122:6828/871482489 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+318 (secure 0 0 0) 0x7fd00c111a70 con 0x7fcff406bcf0 2024-04-16T05:43:46.448 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:43:46.448 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T05:43:46.448 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:43:46.448 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:43:46.448 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [], 2024-04-16T05:43:46.448 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "1/21 daemons upgraded", 2024-04-16T05:43:46.449 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "", 2024-04-16T05:43:46.449 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:43:46.449 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:43:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.451+0000 7fd014400700 1 -- 172.21.15.27:0/1583678478 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fcff406bcf0 msgr2=0x7fcff406e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.451+0000 7fd014400700 1 --2- 172.21.15.27:0/1583678478 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fcff406bcf0 0x7fcff406e1a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcff800b7a0 tx=0x7fcff8009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.451+0000 7fd014400700 1 -- 172.21.15.27:0/1583678478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00c0fce40 msgr2=0x7fd00c10fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.451+0000 7fd014400700 1 --2- 172.21.15.27:0/1583678478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00c0fce40 0x7fd00c10fc50 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd000002a80 tx=0x7fd000004600 comp rx=0 tx=0).stop 2024-04-16T05:43:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.451+0000 7fd014400700 1 -- 172.21.15.27:0/1583678478 shutdown_connections 2024-04-16T05:43:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.451+0000 7fd014400700 1 --2- 172.21.15.27:0/1583678478 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fcff406bcf0 0x7fcff406e1a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.451+0000 7fd014400700 1 --2- 172.21.15.27:0/1583678478 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd00c0fce40 0x7fd00c10fc50 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:46.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.451+0000 7fd014400700 1 --2- 172.21.15.27:0/1583678478 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd00c0fe040 0x7fd00c110190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:46.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.451+0000 7fd014400700 1 -- 172.21.15.27:0/1583678478 >> 172.21.15.27:0/1583678478 conn(0x7fd00c0f8570 msgr2=0x7fd00c0f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:46.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.452+0000 7fd014400700 1 -- 172.21.15.27:0/1583678478 shutdown_connections 2024-04-16T05:43:46.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.452+0000 7fd014400700 1 -- 172.21.15.27:0/1583678478 wait complete. 2024-04-16T05:43:46.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:46 smithi122 ceph-mon[35998]: from='client.14692 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:43:46.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:46 smithi122 ceph-mon[35998]: pgmap v25: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:46.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:46 smithi122 ceph-mon[35998]: from='client.14696 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:43:46.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:46 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/2497354809' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:43:46.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:46 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:43:46.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:46 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/1474701039' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:43:46.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.602+0000 7f49c5400700 1 -- 172.21.15.27:0/4250852547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49c0071ea0 msgr2=0x7f49c00703b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:46.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.602+0000 7f49c5400700 1 --2- 172.21.15.27:0/4250852547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49c0071ea0 0x7f49c00703b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f49b0004940 tx=0x7f49b0009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:46.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.602+0000 7f49c5400700 1 -- 172.21.15.27:0/4250852547 shutdown_connections 2024-04-16T05:43:46.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.602+0000 7f49c5400700 1 --2- 172.21.15.27:0/4250852547 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c00708f0 0x7f49c0070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:46.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.602+0000 7f49c5400700 1 --2- 172.21.15.27:0/4250852547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49c0071ea0 0x7f49c00703b0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:46.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.602+0000 7f49c5400700 1 -- 172.21.15.27:0/4250852547 >> 172.21.15.27:0/4250852547 conn(0x7f49c00f83a0 msgr2=0x7f49c00fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:46.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.602+0000 7f49c5400700 1 -- 172.21.15.27:0/4250852547 shutdown_connections 2024-04-16T05:43:46.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.602+0000 7f49c5400700 1 -- 172.21.15.27:0/4250852547 wait complete. 2024-04-16T05:43:46.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.603+0000 7f49c5400700 1 Processor -- start 2024-04-16T05:43:46.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.603+0000 7f49c5400700 1 -- start start 2024-04-16T05:43:46.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.604+0000 7f49c5400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49c00708f0 0x7f49c010fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:46.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.604+0000 7f49c5400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c0071ea0 0x7f49c0110000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:46.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.604+0000 7f49c5400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c0110600 con 0x7f49c00708f0 2024-04-16T05:43:46.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.604+0000 7f49c5400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c0110e50 con 0x7f49c0071ea0 2024-04-16T05:43:46.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.604+0000 7f49bfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49c00708f0 0x7f49c010fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:46.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.604+0000 7f49bfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49c00708f0 0x7f49c010fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60752/0 (socket says 172.21.15.27:60752) 2024-04-16T05:43:46.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.604+0000 7f49bfe00700 1 -- 172.21.15.27:0/2341348537 learned_addr learned my addr 172.21.15.27:0/2341348537 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:43:46.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.604+0000 7f49bf400700 1 --2- 172.21.15.27:0/2341348537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c0071ea0 0x7f49c0110000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:46.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.604+0000 7f49bfe00700 1 -- 172.21.15.27:0/2341348537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c0071ea0 msgr2=0x7f49c0110000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:46.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.604+0000 7f49bfe00700 1 --2- 172.21.15.27:0/2341348537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c0071ea0 0x7f49c0110000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:46.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.604+0000 7f49bfe00700 1 -- 172.21.15.27:0/2341348537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b0005040 con 0x7f49c00708f0 2024-04-16T05:43:46.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.604+0000 7f49bf400700 1 --2- 172.21.15.27:0/2341348537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c0071ea0 0x7f49c0110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:43:46.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.605+0000 7f49bfe00700 1 --2- 172.21.15.27:0/2341348537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49c00708f0 0x7f49c010fac0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f49b000b970 tx=0x7f49b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:46.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.605+0000 7f49b7e00700 1 -- 172.21.15.27:0/2341348537 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49b000e040 con 0x7f49c00708f0 2024-04-16T05:43:46.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.605+0000 7f49b7e00700 1 -- 172.21.15.27:0/2341348537 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f49b0004290 con 0x7f49c00708f0 2024-04-16T05:43:46.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.605+0000 7f49c5400700 1 -- 172.21.15.27:0/2341348537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c01110d0 con 0x7f49c00708f0 2024-04-16T05:43:46.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.605+0000 7f49b7e00700 1 -- 172.21.15.27:0/2341348537 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49b001c630 con 0x7f49c00708f0 2024-04-16T05:43:46.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.605+0000 7f49c5400700 1 -- 172.21.15.27:0/2341348537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c0111610 con 0x7f49c00708f0 2024-04-16T05:43:46.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.606+0000 7f49c5400700 1 -- 172.21.15.27:0/2341348537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c00fc020 con 0x7f49c00708f0 2024-04-16T05:43:46.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.610+0000 7f49b7e00700 1 -- 172.21.15.27:0/2341348537 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 89652+0+0 (secure 0 0 0) 0x7f49b000bbf0 con 0x7f49c00708f0 2024-04-16T05:43:46.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.611+0000 7f49b7e00700 1 --2- 172.21.15.27:0/2341348537 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f49a406bd00 0x7f49a406e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:43:46.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.611+0000 7f49b7e00700 1 -- 172.21.15.27:0/2341348537 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f49b008bd80 con 0x7f49c00708f0 2024-04-16T05:43:46.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.612+0000 7f49bf400700 1 --2- 172.21.15.27:0/2341348537 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f49a406bd00 0x7f49a406e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:43:46.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.613+0000 7f49bf400700 1 --2- 172.21.15.27:0/2341348537 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f49a406bd00 0x7f49a406e1b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f49ac009fd0 tx=0x7f49ac00b040 comp rx=0 tx=0).ready entity=mgr.24369 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:43:46.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.617+0000 7f49b7e00700 1 -- 172.21.15.27:0/2341348537 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f49b0058f30 con 0x7f49c00708f0 2024-04-16T05:43:46.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.979+0000 7f49c5400700 1 -- 172.21.15.27:0/2341348537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f49c004bed0 con 0x7f49c00708f0 2024-04-16T05:43:46.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.980+0000 7f49b7e00700 1 -- 172.21.15.27:0/2341348537 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f49b0013e60 con 0x7f49c00708f0 2024-04-16T05:43:46.979 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:43:46.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.983+0000 7f49c5400700 1 -- 172.21.15.27:0/2341348537 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f49a406bd00 msgr2=0x7f49a406e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:46.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.983+0000 7f49c5400700 1 --2- 172.21.15.27:0/2341348537 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f49a406bd00 0x7f49a406e1b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f49ac009fd0 tx=0x7f49ac00b040 comp rx=0 tx=0).stop 2024-04-16T05:43:46.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.983+0000 7f49c5400700 1 -- 172.21.15.27:0/2341348537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49c00708f0 msgr2=0x7f49c010fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:43:46.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.983+0000 7f49c5400700 1 --2- 172.21.15.27:0/2341348537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49c00708f0 0x7f49c010fac0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f49b000b970 tx=0x7f49b0009e30 comp rx=0 tx=0).stop 2024-04-16T05:43:46.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.983+0000 7f49c5400700 1 -- 172.21.15.27:0/2341348537 shutdown_connections 2024-04-16T05:43:46.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.983+0000 7f49c5400700 1 --2- 172.21.15.27:0/2341348537 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f49a406bd00 0x7f49a406e1b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:46.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.983+0000 7f49c5400700 1 --2- 172.21.15.27:0/2341348537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49c00708f0 0x7f49c010fac0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:46.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.983+0000 7f49c5400700 1 --2- 172.21.15.27:0/2341348537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49c0071ea0 0x7f49c0110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:43:46.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.983+0000 7f49c5400700 1 -- 172.21.15.27:0/2341348537 >> 172.21.15.27:0/2341348537 conn(0x7f49c00f83a0 msgr2=0x7f49c00f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:43:46.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.984+0000 7f49c5400700 1 -- 172.21.15.27:0/2341348537 shutdown_connections 2024-04-16T05:43:46.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:43:46.984+0000 7f49c5400700 1 -- 172.21.15.27:0/2341348537 wait complete. 2024-04-16T05:43:47.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:47 smithi027 ceph-mon[29206]: from='client.14708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:43:47.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:47 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/2341348537' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:43:47.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:47 smithi122 ceph-mon[35998]: from='client.14708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:43:47.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:47 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/2341348537' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:43:48.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:48 smithi122 ceph-mon[35998]: pgmap v26: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:48.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:48 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:48.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:48 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:48.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:48 smithi027 ceph-mon[29206]: pgmap v26: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:48.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:48 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:48.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:48 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:49.474 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:49 smithi027 ceph-mon[29206]: Reconfiguring prometheus.smithi027 (dependencies changed)... 2024-04-16T05:43:49.475 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:49 smithi027 ceph-mon[29206]: Reconfiguring daemon prometheus.smithi027 on smithi027 2024-04-16T05:43:49.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:49 smithi122 ceph-mon[35998]: Reconfiguring prometheus.smithi027 (dependencies changed)... 2024-04-16T05:43:49.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:49 smithi122 ceph-mon[35998]: Reconfiguring daemon prometheus.smithi027 on smithi027 2024-04-16T05:43:50.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:50 smithi122 ceph-mon[35998]: pgmap v27: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:50.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:50 smithi027 ceph-mon[29206]: pgmap v27: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:52.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:52 smithi122 ceph-mon[35998]: pgmap v28: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:52.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:52 smithi027 ceph-mon[29206]: pgmap v28: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:54.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:54 smithi122 ceph-mon[35998]: pgmap v29: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:54.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:54 smithi027 ceph-mon[29206]: pgmap v29: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:56.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:56 smithi122 ceph-mon[35998]: pgmap v30: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:56.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:56 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:56.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:56 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:56.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:56 smithi027 ceph-mon[29206]: pgmap v30: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:56.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:56 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:56.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:56 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:43:57.420 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:57 smithi122 ceph-mon[35998]: Reconfiguring node-exporter.smithi122 (dependencies changed)... 2024-04-16T05:43:57.420 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:57 smithi122 ceph-mon[35998]: Reconfiguring daemon node-exporter.smithi122 on smithi122 2024-04-16T05:43:57.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:57 smithi027 ceph-mon[29206]: Reconfiguring node-exporter.smithi122 (dependencies changed)... 2024-04-16T05:43:57.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:57 smithi027 ceph-mon[29206]: Reconfiguring daemon node-exporter.smithi122 on smithi122 2024-04-16T05:43:58.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:43:58 smithi122 ceph-mon[35998]: pgmap v31: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:43:58.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:43:58 smithi027 ceph-mon[29206]: pgmap v31: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:00.536 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:00 smithi027 ceph-mon[29206]: pgmap v32: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:00.536 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:00 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:00.536 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:00 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:00.536 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:00 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T05:44:00.537 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:00 smithi027 ceph-mon[29206]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T05:44:00.537 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:00 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T05:44:00.537 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:00 smithi027 ceph-mon[29206]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T05:44:00.537 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:00 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T05:44:00.537 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:00 smithi027 ceph-mon[29206]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T05:44:00.537 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:00 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:44:00.537 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:00 smithi027 ceph-mon[29206]: Upgrade: Need to upgrade myself (mgr.smithi122.sgokkq) 2024-04-16T05:44:00.537 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:00 smithi027 ceph-mon[29206]: Upgrade: Need to upgrade myself (mgr.smithi122.sgokkq) 2024-04-16T05:44:00.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:00 smithi122 ceph-mon[35998]: pgmap v32: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:00.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:00 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:00.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:00 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:00.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:00 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T05:44:00.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:00 smithi122 ceph-mon[35998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T05:44:00.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:00 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T05:44:00.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:00 smithi122 ceph-mon[35998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T05:44:00.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:00 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T05:44:00.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:00 smithi122 ceph-mon[35998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T05:44:00.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:00 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:44:00.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:00 smithi122 ceph-mon[35998]: Upgrade: Need to upgrade myself (mgr.smithi122.sgokkq) 2024-04-16T05:44:00.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:00 smithi122 ceph-mon[35998]: Upgrade: Need to upgrade myself (mgr.smithi122.sgokkq) 2024-04-16T05:44:01.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:01 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:44:01.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:01 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:44:02.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:02 smithi027 ceph-mon[29206]: pgmap v33: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:02.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:02 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:02.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:02 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi027.qgzypi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T05:44:02.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:02 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T05:44:02.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:02 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:44:02.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:02 smithi122 ceph-mon[35998]: pgmap v33: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:02.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:02 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:02.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:02 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi027.qgzypi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T05:44:02.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:02 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T05:44:02.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:02 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:44:03.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:03 smithi027 ceph-mon[29206]: Upgrade: Updating mgr.smithi027.qgzypi 2024-04-16T05:44:03.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:03 smithi027 ceph-mon[29206]: Deploying daemon mgr.smithi027.qgzypi on smithi027 2024-04-16T05:44:03.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:03 smithi122 ceph-mon[35998]: Upgrade: Updating mgr.smithi027.qgzypi 2024-04-16T05:44:03.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:03 smithi122 ceph-mon[35998]: Deploying daemon mgr.smithi027.qgzypi on smithi027 2024-04-16T05:44:04.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:04 smithi027 ceph-mon[29206]: pgmap v34: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:04.632 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:04 smithi122 ceph-mon[35998]: pgmap v34: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:06.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:06 smithi027 ceph-mon[29206]: pgmap v35: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:06.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:06 smithi122 ceph-mon[35998]: pgmap v35: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:08.662 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:08 smithi027 ceph-mon[29206]: pgmap v36: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:08.662 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:08 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:08.662 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:08 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:08.662 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:08 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:44:08.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:08 smithi122 ceph-mon[35998]: pgmap v36: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:08.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:08 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:08.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:08 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:08.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:08 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:44:10.477 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:10 smithi027 ceph-mon[29206]: pgmap v37: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:10.477 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:10 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:10.478 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:10 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:10.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:10 smithi122 ceph-mon[35998]: pgmap v37: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:10.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:10 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:10.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:10 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:12.502 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:12 smithi027 ceph-mon[29206]: pgmap v38: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:12.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:12 smithi122 ceph-mon[35998]: pgmap v38: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:14.507 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:14 smithi027 ceph-mon[29206]: pgmap v39: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:14.507 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:14 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:14.507 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:14 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:14.632 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:14 smithi122 ceph-mon[35998]: pgmap v39: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:14.632 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:14 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:14.632 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:14 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:16.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:16 smithi122 ceph-mon[35998]: pgmap v40: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:16 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:16 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:16 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:44:16.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:16 smithi027 ceph-mon[29206]: pgmap v40: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:16.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:16 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:16.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:16 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:16.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:16 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:44:17.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.150+0000 7fcf5ee00700 1 -- 172.21.15.27:0/3610096213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf580fcff0 msgr2=0x7fcf580fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:17.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.150+0000 7fcf5ee00700 1 --2- 172.21.15.27:0/3610096213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf580fcff0 0x7fcf580fd460 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fcf44004940 tx=0x7fcf44009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:17.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.150+0000 7fcf5ee00700 1 -- 172.21.15.27:0/3610096213 shutdown_connections 2024-04-16T05:44:17.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.150+0000 7fcf5ee00700 1 --2- 172.21.15.27:0/3610096213 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf580fcff0 0x7fcf580fd460 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:17.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.150+0000 7fcf5ee00700 1 --2- 172.21.15.27:0/3610096213 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf580fbd40 0x7fcf580fc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:17.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.150+0000 7fcf5ee00700 1 -- 172.21.15.27:0/3610096213 >> 172.21.15.27:0/3610096213 conn(0x7fcf580f74c0 msgr2=0x7fcf580f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:17.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.150+0000 7fcf5ee00700 1 -- 172.21.15.27:0/3610096213 shutdown_connections 2024-04-16T05:44:17.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.150+0000 7fcf5ee00700 1 -- 172.21.15.27:0/3610096213 wait complete. 2024-04-16T05:44:17.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.152+0000 7fcf5ee00700 1 Processor -- start 2024-04-16T05:44:17.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.152+0000 7fcf5ee00700 1 -- start start 2024-04-16T05:44:17.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.153+0000 7fcf5ee00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf580fbd40 0x7fcf5810dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:17.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.153+0000 7fcf5ee00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf580fcff0 0x7fcf5810e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:17.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.153+0000 7fcf5ee00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5810e610 con 0x7fcf580fcff0 2024-04-16T05:44:17.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.153+0000 7fcf5ee00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5810edc0 con 0x7fcf580fbd40 2024-04-16T05:44:17.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.153+0000 7fcf57400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf580fcff0 0x7fcf5810e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:17.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.153+0000 7fcf57400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf580fcff0 0x7fcf5810e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55422/0 (socket says 172.21.15.27:55422) 2024-04-16T05:44:17.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.153+0000 7fcf57400700 1 -- 172.21.15.27:0/1862460515 learned_addr learned my addr 172.21.15.27:0/1862460515 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:44:17.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.153+0000 7fcf57e00700 1 --2- 172.21.15.27:0/1862460515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf580fbd40 0x7fcf5810dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:17.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.154+0000 7fcf57400700 1 -- 172.21.15.27:0/1862460515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf580fbd40 msgr2=0x7fcf5810dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:17.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.154+0000 7fcf57400700 1 --2- 172.21.15.27:0/1862460515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf580fbd40 0x7fcf5810dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:17.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.154+0000 7fcf57400700 1 -- 172.21.15.27:0/1862460515 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf44005040 con 0x7fcf580fcff0 2024-04-16T05:44:17.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.154+0000 7fcf57400700 1 --2- 172.21.15.27:0/1862460515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf580fcff0 0x7fcf5810e010 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fcf44009150 tx=0x7fcf44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:17.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.154+0000 7fcf4fe00700 1 -- 172.21.15.27:0/1862460515 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf4400e040 con 0x7fcf580fcff0 2024-04-16T05:44:17.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.154+0000 7fcf4fe00700 1 -- 172.21.15.27:0/1862460515 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf4400bbf0 con 0x7fcf580fcff0 2024-04-16T05:44:17.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.154+0000 7fcf5ee00700 1 -- 172.21.15.27:0/1862460515 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf5810f040 con 0x7fcf580fcff0 2024-04-16T05:44:17.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.155+0000 7fcf4fe00700 1 -- 172.21.15.27:0/1862460515 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf4401c630 con 0x7fcf580fcff0 2024-04-16T05:44:17.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.155+0000 7fcf5ee00700 1 -- 172.21.15.27:0/1862460515 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf5810f580 con 0x7fcf580fcff0 2024-04-16T05:44:17.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.156+0000 7fcf5ee00700 1 -- 172.21.15.27:0/1862460515 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf58101b80 con 0x7fcf580fcff0 2024-04-16T05:44:17.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.158+0000 7fcf4fe00700 1 -- 172.21.15.27:0/1862460515 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 89652+0+0 (secure 0 0 0) 0x7fcf4400bd60 con 0x7fcf580fcff0 2024-04-16T05:44:17.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.160+0000 7fcf4fe00700 1 --2- 172.21.15.27:0/1862460515 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fcf3c06bcf0 0x7fcf3c06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:17.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.160+0000 7fcf4fe00700 1 -- 172.21.15.27:0/1862460515 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7fcf4408bf40 con 0x7fcf580fcff0 2024-04-16T05:44:17.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.160+0000 7fcf57e00700 1 --2- 172.21.15.27:0/1862460515 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fcf3c06bcf0 0x7fcf3c06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:17.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.162+0000 7fcf57e00700 1 --2- 172.21.15.27:0/1862460515 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fcf3c06bcf0 0x7fcf3c06e1a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcf48004990 tx=0x7fcf48009ee0 comp rx=0 tx=0).ready entity=mgr.24369 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:17.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.164+0000 7fcf4fe00700 1 -- 172.21.15.27:0/1862460515 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fcf44058fb0 con 0x7fcf580fcff0 2024-04-16T05:44:17.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.418+0000 7fcf5ee00700 1 -- 172.21.15.27:0/1862460515 --> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf5810f7a0 con 0x7fcf3c06bcf0 2024-04-16T05:44:17.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.421+0000 7fcf4fe00700 1 -- 172.21.15.27:0/1862460515 <== mgr.24369 v2:172.21.15.122:6828/871482489 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7fcf5810f7a0 con 0x7fcf3c06bcf0 2024-04-16T05:44:17.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.424+0000 7fcf5ee00700 1 -- 172.21.15.27:0/1862460515 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fcf3c06bcf0 msgr2=0x7fcf3c06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:17.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.424+0000 7fcf5ee00700 1 --2- 172.21.15.27:0/1862460515 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fcf3c06bcf0 0x7fcf3c06e1a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcf48004990 tx=0x7fcf48009ee0 comp rx=0 tx=0).stop 2024-04-16T05:44:17.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.424+0000 7fcf5ee00700 1 -- 172.21.15.27:0/1862460515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf580fcff0 msgr2=0x7fcf5810e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:17.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.424+0000 7fcf5ee00700 1 --2- 172.21.15.27:0/1862460515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf580fcff0 0x7fcf5810e010 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fcf44009150 tx=0x7fcf44009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:17.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.424+0000 7fcf5ee00700 1 -- 172.21.15.27:0/1862460515 shutdown_connections 2024-04-16T05:44:17.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.424+0000 7fcf5ee00700 1 --2- 172.21.15.27:0/1862460515 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf580fbd40 0x7fcf5810dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:17.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.424+0000 7fcf5ee00700 1 --2- 172.21.15.27:0/1862460515 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7fcf3c06bcf0 0x7fcf3c06e1a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:17.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.424+0000 7fcf5ee00700 1 --2- 172.21.15.27:0/1862460515 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf580fcff0 0x7fcf5810e010 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:17.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.424+0000 7fcf5ee00700 1 -- 172.21.15.27:0/1862460515 >> 172.21.15.27:0/1862460515 conn(0x7fcf580f74c0 msgr2=0x7fcf58065b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:17.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.424+0000 7fcf5ee00700 1 -- 172.21.15.27:0/1862460515 shutdown_connections 2024-04-16T05:44:17.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.424+0000 7fcf5ee00700 1 -- 172.21.15.27:0/1862460515 wait complete. 2024-04-16T05:44:17.439 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:44:17.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.588+0000 7ff29fc00700 1 -- 172.21.15.27:0/98951387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2980fe040 msgr2=0x7ff2980fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:17.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.588+0000 7ff29fc00700 1 --2- 172.21.15.27:0/98951387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2980fe040 0x7ff2980fe4b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff284004940 tx=0x7ff284009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:17.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.589+0000 7ff29fc00700 1 -- 172.21.15.27:0/98951387 shutdown_connections 2024-04-16T05:44:17.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.589+0000 7ff29fc00700 1 --2- 172.21.15.27:0/98951387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2980fe040 0x7ff2980fe4b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:17.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.589+0000 7ff29fc00700 1 --2- 172.21.15.27:0/98951387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2980fce40 0x7ff2980fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:17.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.589+0000 7ff29fc00700 1 -- 172.21.15.27:0/98951387 >> 172.21.15.27:0/98951387 conn(0x7ff2980f8530 msgr2=0x7ff2980fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:17.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.589+0000 7ff29fc00700 1 -- 172.21.15.27:0/98951387 shutdown_connections 2024-04-16T05:44:17.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.589+0000 7ff29fc00700 1 -- 172.21.15.27:0/98951387 wait complete. 2024-04-16T05:44:17.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.590+0000 7ff29fc00700 1 Processor -- start 2024-04-16T05:44:17.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.590+0000 7ff29fc00700 1 -- start start 2024-04-16T05:44:17.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.590+0000 7ff29fc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2980fce40 0x7ff29810fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.590+0000 7ff29fc00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2980fe040 0x7ff298110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.590+0000 7ff29fc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff298110850 con 0x7ff2980fce40 2024-04-16T05:44:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.590+0000 7ff29fc00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff298110f90 con 0x7ff2980fe040 2024-04-16T05:44:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.591+0000 7ff29d200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2980fce40 0x7ff29810fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.591+0000 7ff29d200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2980fce40 0x7ff29810fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55442/0 (socket says 172.21.15.27:55442) 2024-04-16T05:44:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.591+0000 7ff29d200700 1 -- 172.21.15.27:0/52737831 learned_addr learned my addr 172.21.15.27:0/52737831 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:44:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.591+0000 7ff297e00700 1 --2- 172.21.15.27:0/52737831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2980fe040 0x7ff298110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.591+0000 7ff29d200700 1 -- 172.21.15.27:0/52737831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2980fe040 msgr2=0x7ff298110250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.591+0000 7ff29d200700 1 --2- 172.21.15.27:0/52737831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2980fe040 0x7ff298110250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:17.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.591+0000 7ff29d200700 1 -- 172.21.15.27:0/52737831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff284005040 con 0x7ff2980fce40 2024-04-16T05:44:17.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.592+0000 7ff29d200700 1 --2- 172.21.15.27:0/52737831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2980fce40 0x7ff29810fd10 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff28c002a80 tx=0x7ff28c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:17.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.592+0000 7ff294c00700 1 -- 172.21.15.27:0/52737831 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff28c007cc0 con 0x7ff2980fce40 2024-04-16T05:44:17.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.592+0000 7ff294c00700 1 -- 172.21.15.27:0/52737831 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff28c00f040 con 0x7ff2980fce40 2024-04-16T05:44:17.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.592+0000 7ff294c00700 1 -- 172.21.15.27:0/52737831 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff28c00dad0 con 0x7ff2980fce40 2024-04-16T05:44:17.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.592+0000 7ff29fc00700 1 -- 172.21.15.27:0/52737831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff298111270 con 0x7ff2980fce40 2024-04-16T05:44:17.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.592+0000 7ff29fc00700 1 -- 172.21.15.27:0/52737831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff298111830 con 0x7ff2980fce40 2024-04-16T05:44:17.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.594+0000 7ff29fc00700 1 -- 172.21.15.27:0/52737831 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2981019d0 con 0x7ff2980fce40 2024-04-16T05:44:17.594 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.594+0000 7ff294c00700 1 -- 172.21.15.27:0/52737831 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 89652+0+0 (secure 0 0 0) 0x7ff28c00dc30 con 0x7ff2980fce40 2024-04-16T05:44:17.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.595+0000 7ff294c00700 1 --2- 172.21.15.27:0/52737831 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7ff28006bcf0 0x7ff28006e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:17.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.596+0000 7ff294c00700 1 -- 172.21.15.27:0/52737831 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7ff28c08a840 con 0x7ff2980fce40 2024-04-16T05:44:17.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.596+0000 7ff297e00700 1 --2- 172.21.15.27:0/52737831 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7ff28006bcf0 0x7ff28006e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:17.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.597+0000 7ff297e00700 1 --2- 172.21.15.27:0/52737831 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7ff28006bcf0 0x7ff28006e1a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff28400b3b0 tx=0x7ff284009e30 comp rx=0 tx=0).ready entity=mgr.24369 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:17.599 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.600+0000 7ff294c00700 1 -- 172.21.15.27:0/52737831 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7ff28c0578b0 con 0x7ff2980fce40 2024-04-16T05:44:17.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.860+0000 7ff29fc00700 1 -- 172.21.15.27:0/52737831 --> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff298111aa0 con 0x7ff28006bcf0 2024-04-16T05:44:17.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.863+0000 7ff294c00700 1 -- 172.21.15.27:0/52737831 <== mgr.24369 v2:172.21.15.122:6828/871482489 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7ff298111aa0 con 0x7ff28006bcf0 2024-04-16T05:44:17.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.866+0000 7ff29fc00700 1 -- 172.21.15.27:0/52737831 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7ff28006bcf0 msgr2=0x7ff28006e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:17.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.866+0000 7ff29fc00700 1 --2- 172.21.15.27:0/52737831 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7ff28006bcf0 0x7ff28006e1a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff28400b3b0 tx=0x7ff284009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:17.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.866+0000 7ff29fc00700 1 -- 172.21.15.27:0/52737831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2980fce40 msgr2=0x7ff29810fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:17.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.866+0000 7ff29fc00700 1 --2- 172.21.15.27:0/52737831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2980fce40 0x7ff29810fd10 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff28c002a80 tx=0x7ff28c004600 comp rx=0 tx=0).stop 2024-04-16T05:44:17.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.866+0000 7ff29fc00700 1 -- 172.21.15.27:0/52737831 shutdown_connections 2024-04-16T05:44:17.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.866+0000 7ff29fc00700 1 --2- 172.21.15.27:0/52737831 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7ff28006bcf0 0x7ff28006e1a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:17.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.866+0000 7ff29fc00700 1 --2- 172.21.15.27:0/52737831 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2980fce40 0x7ff29810fd10 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:17.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.866+0000 7ff29fc00700 1 --2- 172.21.15.27:0/52737831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff2980fe040 0x7ff298110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:17.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.866+0000 7ff29fc00700 1 -- 172.21.15.27:0/52737831 >> 172.21.15.27:0/52737831 conn(0x7ff2980f8530 msgr2=0x7ff2980f9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:17.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.866+0000 7ff29fc00700 1 -- 172.21.15.27:0/52737831 shutdown_connections 2024-04-16T05:44:17.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:17.866+0000 7ff29fc00700 1 -- 172.21.15.27:0/52737831 wait complete. 2024-04-16T05:44:18.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.014+0000 7f2863200700 1 -- 172.21.15.27:0/104227070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f285c071f50 msgr2=0x7f285c070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:18.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.014+0000 7f2863200700 1 --2- 172.21.15.27:0/104227070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f285c071f50 0x7f285c070460 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f284c009ce0 tx=0x7f284c004d10 comp rx=0 tx=0).stop 2024-04-16T05:44:18.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.014+0000 7f2863200700 1 -- 172.21.15.27:0/104227070 shutdown_connections 2024-04-16T05:44:18.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.015+0000 7f2863200700 1 --2- 172.21.15.27:0/104227070 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f285c0709a0 0x7f285c070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:18.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.015+0000 7f2863200700 1 --2- 172.21.15.27:0/104227070 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f285c071f50 0x7f285c070460 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:18.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.015+0000 7f2863200700 1 -- 172.21.15.27:0/104227070 >> 172.21.15.27:0/104227070 conn(0x7f285c0f8530 msgr2=0x7f285c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:18.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.015+0000 7f2863200700 1 -- 172.21.15.27:0/104227070 shutdown_connections 2024-04-16T05:44:18.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.015+0000 7f2863200700 1 -- 172.21.15.27:0/104227070 wait complete. 2024-04-16T05:44:18.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.016+0000 7f2863200700 1 Processor -- start 2024-04-16T05:44:18.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.016+0000 7f2863200700 1 -- start start 2024-04-16T05:44:18.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.016+0000 7f2863200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f285c0709a0 0x7f285c114230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:18.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.016+0000 7f2863200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f285c071f50 0x7f285c114770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:18.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.016+0000 7f2863200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285c114d70 con 0x7f285c0709a0 2024-04-16T05:44:18.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.016+0000 7f2863200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285c1155c0 con 0x7f285c071f50 2024-04-16T05:44:18.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.017+0000 7f285be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f285c0709a0 0x7f285c114230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:18.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.017+0000 7f285be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f285c0709a0 0x7f285c114230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55456/0 (socket says 172.21.15.27:55456) 2024-04-16T05:44:18.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.017+0000 7f285be00700 1 -- 172.21.15.27:0/2261706078 learned_addr learned my addr 172.21.15.27:0/2261706078 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:44:18.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.017+0000 7f285b400700 1 --2- 172.21.15.27:0/2261706078 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f285c071f50 0x7f285c114770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:18.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.017+0000 7f285be00700 1 -- 172.21.15.27:0/2261706078 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f285c071f50 msgr2=0x7f285c114770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:18.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.017+0000 7f285be00700 1 --2- 172.21.15.27:0/2261706078 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f285c071f50 0x7f285c114770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:18.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.017+0000 7f285be00700 1 -- 172.21.15.27:0/2261706078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284c009940 con 0x7f285c0709a0 2024-04-16T05:44:18.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.018+0000 7f285be00700 1 --2- 172.21.15.27:0/2261706078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f285c0709a0 0x7f285c114230 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f284c009910 tx=0x7f284c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:18.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.018+0000 7f2853e00700 1 -- 172.21.15.27:0/2261706078 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f284c00e040 con 0x7f285c0709a0 2024-04-16T05:44:18.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.018+0000 7f2853e00700 1 -- 172.21.15.27:0/2261706078 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f284c008ed0 con 0x7f285c0709a0 2024-04-16T05:44:18.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.018+0000 7f2853e00700 1 -- 172.21.15.27:0/2261706078 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f284c01c800 con 0x7f285c0709a0 2024-04-16T05:44:18.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.018+0000 7f2863200700 1 -- 172.21.15.27:0/2261706078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f285c115840 con 0x7f285c0709a0 2024-04-16T05:44:18.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.018+0000 7f2863200700 1 -- 172.21.15.27:0/2261706078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f285c115d80 con 0x7f285c0709a0 2024-04-16T05:44:18.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.020+0000 7f2863200700 1 -- 172.21.15.27:0/2261706078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f285c05f5f0 con 0x7f285c0709a0 2024-04-16T05:44:18.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.021+0000 7f2853e00700 1 -- 172.21.15.27:0/2261706078 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 89652+0+0 (secure 0 0 0) 0x7f284c012410 con 0x7f285c0709a0 2024-04-16T05:44:18.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.021+0000 7f2853e00700 1 --2- 172.21.15.27:0/2261706078 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f284006bcf0 0x7f284006e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:18.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.022+0000 7f2853e00700 1 -- 172.21.15.27:0/2261706078 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f284c08c870 con 0x7f285c0709a0 2024-04-16T05:44:18.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.022+0000 7f285b400700 1 --2- 172.21.15.27:0/2261706078 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f284006bcf0 0x7f284006e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:18.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.023+0000 7f285b400700 1 --2- 172.21.15.27:0/2261706078 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f284006bcf0 0x7f284006e1a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2848007fd0 tx=0x7f2848009040 comp rx=0 tx=0).ready entity=mgr.24369 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:18.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.026+0000 7f2853e00700 1 -- 172.21.15.27:0/2261706078 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f284c059860 con 0x7f285c0709a0 2024-04-16T05:44:18.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.255+0000 7f2863200700 1 -- 172.21.15.27:0/2261706078 --> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f285c116050 con 0x7f284006bcf0 2024-04-16T05:44:18.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.285+0000 7f2853e00700 1 -- 172.21.15.27:0/2261706078 <== mgr.24369 v2:172.21.15.122:6828/871482489 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f285c116050 con 0x7f284006bcf0 2024-04-16T05:44:18.285 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:44:18.285 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (38s) 2s ago 11m 13.4M - 0.25.0 c8568f914cd2 4ed63a8a6b33 2024-04-16T05:44:18.285 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (11m) 2s ago 11m 6665k - 17.2.7-847-g45317f95 b333581ccdf5 f1acb6959c63 2024-04-16T05:44:18.285 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (10m) 4s ago 10m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 e81efc20e8c0 2024-04-16T05:44:18.285 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (31s) 2s ago 10m 73.7M - 9.4.7 954c08fa6188 f7ee2a1afd12 2024-04-16T05:44:18.285 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (3m) 2s ago 3m 22.9M - 17.2.7-847-g45317f95 b333581ccdf5 0824e2621be5 2024-04-16T05:44:18.285 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (4m) 2s ago 4m 12.1M - 17.2.7-847-g45317f95 b333581ccdf5 b332a72cc594 2024-04-16T05:44:18.285 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (3m) 4s ago 3m 12.2M - 17.2.7-847-g45317f95 b333581ccdf5 f5e36815ddfa 2024-04-16T05:44:18.285 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (3m) 4s ago 3m 12.2M - 17.2.7-847-g45317f95 b333581ccdf5 755a73c65a7f 2024-04-16T05:44:18.285 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:8443,9283,8765 running (10s) 2s ago 13m 70.4M - 19.0.0-2884-g218a6914 a4e4811815e4 b69a0b08e12b 2024-04-16T05:44:18.286 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283 running (89s) 4s ago 10m 503M - 19.0.0-2884-g218a6914 a4e4811815e4 b1a1b7a6d536 2024-04-16T05:44:18.286 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (13m) 2s ago 13m 48.8M 2048M 17.2.7-847-g45317f95 b333581ccdf5 88575dbb8d98 2024-04-16T05:44:18.286 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (9m) 4s ago 9m 38.7M 2048M 17.2.7-847-g45317f95 b333581ccdf5 5df19eab6d21 2024-04-16T05:44:18.286 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (45s) 2s ago 11m 7025k - 1.5.0 0da6a335fe13 5e29e29868b0 2024-04-16T05:44:18.286 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (19s) 4s ago 10m 6786k - 1.5.0 0da6a335fe13 a360922bc621 2024-04-16T05:44:18.286 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (8m) 2s ago 8m 46.6M 1557M 17.2.7-847-g45317f95 b333581ccdf5 15e049573bca 2024-04-16T05:44:18.286 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (8m) 2s ago 8m 46.5M 1557M 17.2.7-847-g45317f95 b333581ccdf5 e146295f6500 2024-04-16T05:44:18.286 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (7m) 2s ago 7m 47.6M 1557M 17.2.7-847-g45317f95 b333581ccdf5 6212f6a18fa4 2024-04-16T05:44:18.286 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (7m) 4s ago 7m 45.1M 2581M 17.2.7-847-g45317f95 b333581ccdf5 d6724ad05f9f 2024-04-16T05:44:18.286 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (6m) 4s ago 6m 43.3M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7a59908a3e10 2024-04-16T05:44:18.286 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (5m) 4s ago 5m 42.7M 2581M 17.2.7-847-g45317f95 b333581ccdf5 8e4b60163907 2024-04-16T05:44:18.286 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (22s) 2s ago 10m 44.0M - 2.43.0 a07b618ecd1d f22b85a76bfc 2024-04-16T05:44:18.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.288+0000 7f2863200700 1 -- 172.21.15.27:0/2261706078 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f284006bcf0 msgr2=0x7f284006e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:18.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.288+0000 7f2863200700 1 --2- 172.21.15.27:0/2261706078 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f284006bcf0 0x7f284006e1a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2848007fd0 tx=0x7f2848009040 comp rx=0 tx=0).stop 2024-04-16T05:44:18.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.288+0000 7f2863200700 1 -- 172.21.15.27:0/2261706078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f285c0709a0 msgr2=0x7f285c114230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:18.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.288+0000 7f2863200700 1 --2- 172.21.15.27:0/2261706078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f285c0709a0 0x7f285c114230 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f284c009910 tx=0x7f284c004d10 comp rx=0 tx=0).stop 2024-04-16T05:44:18.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.288+0000 7f2863200700 1 -- 172.21.15.27:0/2261706078 shutdown_connections 2024-04-16T05:44:18.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.288+0000 7f2863200700 1 --2- 172.21.15.27:0/2261706078 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f284006bcf0 0x7f284006e1a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:18.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.288+0000 7f2863200700 1 --2- 172.21.15.27:0/2261706078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f285c0709a0 0x7f285c114230 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:18.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.288+0000 7f2863200700 1 --2- 172.21.15.27:0/2261706078 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f285c071f50 0x7f285c114770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:18.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.288+0000 7f2863200700 1 -- 172.21.15.27:0/2261706078 >> 172.21.15.27:0/2261706078 conn(0x7f285c0f8530 msgr2=0x7f285c0f9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:18.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.288+0000 7f2863200700 1 -- 172.21.15.27:0/2261706078 shutdown_connections 2024-04-16T05:44:18.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.289+0000 7f2863200700 1 -- 172.21.15.27:0/2261706078 wait complete. 2024-04-16T05:44:18.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.441+0000 7f08a0400700 1 -- 172.21.15.27:0/3920986613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08981046d0 msgr2=0x7f0898104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:18.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.441+0000 7f08a0400700 1 --2- 172.21.15.27:0/3920986613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08981046d0 0x7f0898104b40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0884004940 tx=0x7f0884009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:18.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.441+0000 7f08a0400700 1 -- 172.21.15.27:0/3920986613 shutdown_connections 2024-04-16T05:44:18.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.441+0000 7f08a0400700 1 --2- 172.21.15.27:0/3920986613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08981046d0 0x7f0898104b40 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:18.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.441+0000 7f08a0400700 1 --2- 172.21.15.27:0/3920986613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0898103420 0x7f0898103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:18.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.441+0000 7f08a0400700 1 -- 172.21.15.27:0/3920986613 >> 172.21.15.27:0/3920986613 conn(0x7f0898072640 msgr2=0x7f0898074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:18.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.441+0000 7f08a0400700 1 -- 172.21.15.27:0/3920986613 shutdown_connections 2024-04-16T05:44:18.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.441+0000 7f08a0400700 1 -- 172.21.15.27:0/3920986613 wait complete. 2024-04-16T05:44:18.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.442+0000 7f08a0400700 1 Processor -- start 2024-04-16T05:44:18.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.442+0000 7f08a0400700 1 -- start start 2024-04-16T05:44:18.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.442+0000 7f08a0400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0898103420 0x7f0898113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:18.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.442+0000 7f08a0400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08981046d0 0x7f0898114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:18.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.442+0000 7f08a0400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0898114b20 con 0x7f0898103420 2024-04-16T05:44:18.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.442+0000 7f08a0400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08981194c0 con 0x7f08981046d0 2024-04-16T05:44:18.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.442+0000 7f089da00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0898103420 0x7f0898113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:18.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.442+0000 7f089da00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0898103420 0x7f0898113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55482/0 (socket says 172.21.15.27:55482) 2024-04-16T05:44:18.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.442+0000 7f089da00700 1 -- 172.21.15.27:0/3750794336 learned_addr learned my addr 172.21.15.27:0/3750794336 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:44:18.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.443+0000 7f089d000700 1 --2- 172.21.15.27:0/3750794336 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08981046d0 0x7f0898114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:18.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.443+0000 7f089da00700 1 -- 172.21.15.27:0/3750794336 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08981046d0 msgr2=0x7f0898114490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:18.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.443+0000 7f089da00700 1 --2- 172.21.15.27:0/3750794336 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08981046d0 0x7f0898114490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:18.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.443+0000 7f089da00700 1 -- 172.21.15.27:0/3750794336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0884005040 con 0x7f0898103420 2024-04-16T05:44:18.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.443+0000 7f089da00700 1 --2- 172.21.15.27:0/3750794336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0898103420 0x7f0898113f50 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f088c00b790 tx=0x7f088c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:18.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.443+0000 7f0895600700 1 -- 172.21.15.27:0/3750794336 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f088c00f770 con 0x7f0898103420 2024-04-16T05:44:18.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.443+0000 7f0895600700 1 -- 172.21.15.27:0/3750794336 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f088c00fda0 con 0x7f0898103420 2024-04-16T05:44:18.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.444+0000 7f0895600700 1 -- 172.21.15.27:0/3750794336 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f088c00d310 con 0x7f0898103420 2024-04-16T05:44:18.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.444+0000 7f08a0400700 1 -- 172.21.15.27:0/3750794336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0898119720 con 0x7f0898103420 2024-04-16T05:44:18.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.444+0000 7f08a0400700 1 -- 172.21.15.27:0/3750794336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0898119c90 con 0x7f0898103420 2024-04-16T05:44:18.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.445+0000 7f08a0400700 1 -- 172.21.15.27:0/3750794336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f089804c710 con 0x7f0898103420 2024-04-16T05:44:18.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.446+0000 7f0895600700 1 -- 172.21.15.27:0/3750794336 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 25) v1 ==== 89652+0+0 (secure 0 0 0) 0x7f088c010420 con 0x7f0898103420 2024-04-16T05:44:18.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.447+0000 7f0895600700 1 --2- 172.21.15.27:0/3750794336 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f088006bd00 0x7f088006e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:18.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.447+0000 7f0895600700 1 -- 172.21.15.27:0/3750794336 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f088c08a980 con 0x7f0898103420 2024-04-16T05:44:18.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.448+0000 7f089d000700 1 --2- 172.21.15.27:0/3750794336 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f088006bd00 0x7f088006e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:18.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.449+0000 7f089d000700 1 --2- 172.21.15.27:0/3750794336 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f088006bd00 0x7f088006e1b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f088400b3b0 tx=0x7f0884009e30 comp rx=0 tx=0).ready entity=mgr.24369 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:18.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.451+0000 7f0895600700 1 -- 172.21.15.27:0/3750794336 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f088c057b80 con 0x7f0898103420 2024-04-16T05:44:18.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.577+0000 7f0895600700 1 -- 172.21.15.27:0/3750794336 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 26) v1 ==== 92571+0+0 (secure 0 0 0) 0x7f088c054420 con 0x7f0898103420 2024-04-16T05:44:18.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.777+0000 7f08a0400700 1 -- 172.21.15.27:0/3750794336 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0898119f60 con 0x7f0898103420 2024-04-16T05:44:18.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.778+0000 7f0895600700 1 -- 172.21.15.27:0/3750794336 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f088c060be0 con 0x7f0898103420 2024-04-16T05:44:18.777 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:44:18.777 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:44:18.777 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T05:44:18.777 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:44:18.777 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:44:18.778 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:44:18.778 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:44:18.778 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:44:18.778 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T05:44:18.778 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:44:18.778 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:44:18.778 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T05:44:18.778 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:44:18.778 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:44:18.778 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 12, 2024-04-16T05:44:18.778 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:44:18.778 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:44:18.778 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:44:18.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.780+0000 7f08a0400700 1 -- 172.21.15.27:0/3750794336 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f088006bd00 msgr2=0x7f088006e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:18.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.780+0000 7f08a0400700 1 --2- 172.21.15.27:0/3750794336 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f088006bd00 0x7f088006e1b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f088400b3b0 tx=0x7f0884009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:18.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.780+0000 7f08a0400700 1 -- 172.21.15.27:0/3750794336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0898103420 msgr2=0x7f0898113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:18.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.780+0000 7f08a0400700 1 --2- 172.21.15.27:0/3750794336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0898103420 0x7f0898113f50 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f088c00b790 tx=0x7f088c009250 comp rx=0 tx=0).stop 2024-04-16T05:44:18.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.780+0000 7f08a0400700 1 -- 172.21.15.27:0/3750794336 shutdown_connections 2024-04-16T05:44:18.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.780+0000 7f08a0400700 1 --2- 172.21.15.27:0/3750794336 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f088006bd00 0x7f088006e1b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:18.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.780+0000 7f08a0400700 1 --2- 172.21.15.27:0/3750794336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0898103420 0x7f0898113f50 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:18.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.780+0000 7f08a0400700 1 --2- 172.21.15.27:0/3750794336 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08981046d0 0x7f0898114490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:18.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.780+0000 7f08a0400700 1 -- 172.21.15.27:0/3750794336 >> 172.21.15.27:0/3750794336 conn(0x7f0898072640 msgr2=0x7f0898073ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:18.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.781+0000 7f08a0400700 1 -- 172.21.15.27:0/3750794336 shutdown_connections 2024-04-16T05:44:18.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.781+0000 7f08a0400700 1 -- 172.21.15.27:0/3750794336 wait complete. 2024-04-16T05:44:18.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:18 smithi027 ceph-mon[29206]: pgmap v41: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:18.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:18 smithi027 ceph-mon[29206]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:44:18.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:18 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:18.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:18 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:18.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:18 smithi027 ceph-mon[29206]: Standby manager daemon smithi027.qgzypi restarted 2024-04-16T05:44:18.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:18 smithi027 ceph-mon[29206]: Standby manager daemon smithi027.qgzypi started 2024-04-16T05:44:18.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:18 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.qgzypi/crt"}]: dispatch 2024-04-16T05:44:18.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:18 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T05:44:18.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:18 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.qgzypi/key"}]: dispatch 2024-04-16T05:44:18.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:18 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T05:44:18.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.926+0000 7f22ecc00700 1 -- 172.21.15.27:0/1598765022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22e80708f0 msgr2=0x7f22e8070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:18.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.926+0000 7f22ecc00700 1 --2- 172.21.15.27:0/1598765022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22e80708f0 0x7f22e8070d60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f22d4004940 tx=0x7f22d4009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:18.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.926+0000 7f22ecc00700 1 -- 172.21.15.27:0/1598765022 shutdown_connections 2024-04-16T05:44:18.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.926+0000 7f22ecc00700 1 --2- 172.21.15.27:0/1598765022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22e80708f0 0x7f22e8070d60 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:18.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.926+0000 7f22ecc00700 1 --2- 172.21.15.27:0/1598765022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22e8071ea0 0x7f22e80703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:18.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.926+0000 7f22ecc00700 1 -- 172.21.15.27:0/1598765022 >> 172.21.15.27:0/1598765022 conn(0x7f22e80f8530 msgr2=0x7f22e80fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:18.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.926+0000 7f22ecc00700 1 -- 172.21.15.27:0/1598765022 shutdown_connections 2024-04-16T05:44:18.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.926+0000 7f22ecc00700 1 -- 172.21.15.27:0/1598765022 wait complete. 2024-04-16T05:44:18.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.927+0000 7f22ecc00700 1 Processor -- start 2024-04-16T05:44:18.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.927+0000 7f22ecc00700 1 -- start start 2024-04-16T05:44:18.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.927+0000 7f22ecc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22e80708f0 0x7f22e8114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:18.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.927+0000 7f22ecc00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22e8071ea0 0x7f22e8114580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:18.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.927+0000 7f22ecc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e8114c10 con 0x7f22e80708f0 2024-04-16T05:44:18.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.927+0000 7f22ecc00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e8115400 con 0x7f22e8071ea0 2024-04-16T05:44:18.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.928+0000 7f22e5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22e80708f0 0x7f22e8114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:18.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.928+0000 7f22e5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22e80708f0 0x7f22e8114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55492/0 (socket says 172.21.15.27:55492) 2024-04-16T05:44:18.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.928+0000 7f22e5e00700 1 -- 172.21.15.27:0/2209390132 learned_addr learned my addr 172.21.15.27:0/2209390132 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:44:18.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.928+0000 7f22e5400700 1 --2- 172.21.15.27:0/2209390132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22e8071ea0 0x7f22e8114580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:18.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.928+0000 7f22e5e00700 1 -- 172.21.15.27:0/2209390132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22e8071ea0 msgr2=0x7f22e8114580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:18.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.928+0000 7f22e5e00700 1 --2- 172.21.15.27:0/2209390132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22e8071ea0 0x7f22e8114580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:18.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.928+0000 7f22e5e00700 1 -- 172.21.15.27:0/2209390132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d4005040 con 0x7f22e80708f0 2024-04-16T05:44:18.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.928+0000 7f22e5400700 1 --2- 172.21.15.27:0/2209390132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22e8071ea0 0x7f22e8114580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:44:18.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.928+0000 7f22e5e00700 1 --2- 172.21.15.27:0/2209390132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22e80708f0 0x7f22e8114040 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f22d800b790 tx=0x7f22d8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:18.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.929+0000 7f22de000700 1 -- 172.21.15.27:0/2209390132 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22d800f6f0 con 0x7f22e80708f0 2024-04-16T05:44:18.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.929+0000 7f22de000700 1 -- 172.21.15.27:0/2209390132 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f22d800fd20 con 0x7f22e80708f0 2024-04-16T05:44:18.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.929+0000 7f22de000700 1 -- 172.21.15.27:0/2209390132 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22d80103a0 con 0x7f22e80708f0 2024-04-16T05:44:18.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.929+0000 7f22ecc00700 1 -- 172.21.15.27:0/2209390132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22e81156e0 con 0x7f22e80708f0 2024-04-16T05:44:18.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.929+0000 7f22ecc00700 1 -- 172.21.15.27:0/2209390132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22e8115ca0 con 0x7f22e80708f0 2024-04-16T05:44:18.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.930+0000 7f22ecc00700 1 -- 172.21.15.27:0/2209390132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22e805f5f0 con 0x7f22e80708f0 2024-04-16T05:44:18.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.931+0000 7f22de000700 1 -- 172.21.15.27:0/2209390132 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 92571+0+0 (secure 0 0 0) 0x7f22d8011400 con 0x7f22e80708f0 2024-04-16T05:44:18.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.932+0000 7f22de000700 1 --2- 172.21.15.27:0/2209390132 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f22cc06e620 0x7f22cc070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:18.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.932+0000 7f22de000700 1 -- 172.21.15.27:0/2209390132 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f22d808e7d0 con 0x7f22e80708f0 2024-04-16T05:44:18.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.933+0000 7f22e5400700 1 --2- 172.21.15.27:0/2209390132 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f22cc06e620 0x7f22cc070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:18.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.934+0000 7f22e5400700 1 --2- 172.21.15.27:0/2209390132 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f22cc06e620 0x7f22cc070ad0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f22d400b7a0 tx=0x7f22d4009e30 comp rx=0 tx=0).ready entity=mgr.24369 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:18.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:18.936+0000 7f22de000700 1 -- 172.21.15.27:0/2209390132 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f22d805b9d0 con 0x7f22e80708f0 2024-04-16T05:44:19.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:18 smithi122 ceph-mon[35998]: pgmap v41: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:19.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:18 smithi122 ceph-mon[35998]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:44:19.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:18 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:19.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:18 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:19.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:18 smithi122 ceph-mon[35998]: Standby manager daemon smithi027.qgzypi restarted 2024-04-16T05:44:19.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:18 smithi122 ceph-mon[35998]: Standby manager daemon smithi027.qgzypi started 2024-04-16T05:44:19.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:18 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.qgzypi/crt"}]: dispatch 2024-04-16T05:44:19.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:18 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T05:44:19.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:18 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi027.qgzypi/key"}]: dispatch 2024-04-16T05:44:19.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:18 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T05:44:19.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.205+0000 7f22ecc00700 1 -- 172.21.15.27:0/2209390132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f22e810bf90 con 0x7f22e80708f0 2024-04-16T05:44:19.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.206+0000 7f22de000700 1 -- 172.21.15.27:0/2209390132 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1810 (secure 0 0 0) 0x7f22d805b560 con 0x7f22e80708f0 2024-04-16T05:44:19.206 INFO:teuthology.orchestra.run.smithi027.stdout:e16 2024-04-16T05:44:19.206 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:44:19.207 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:44:19.207 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:44:19.207 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:44:19.207 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:44:19.207 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:44:19.207 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 15 2024-04-16T05:44:19.207 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:44:19.207 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:44:19.207 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:40:32.701649+0000 2024-04-16T05:44:19.207 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:44:19.207 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:44:19.207 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:44:19.207 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:44:19.208 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:44:19.208 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:44:19.208 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:44:19.208 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 46 2024-04-16T05:44:19.208 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:44:19.208 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:44:19.208 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:44:19.208 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=14462} 2024-04-16T05:44:19.208 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:44:19.208 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:44:19.208 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:44:19.208 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:44:19.208 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:44:19.208 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:44:19.209 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:44:19.209 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:44:19.209 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:44:19.209 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:44:19.209 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:44:19.209 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:44:19.209 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:44:19.209 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{-1:14474} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2065690722,v1:172.21.15.27:6827/2065690722] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:44:19.209 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:24263} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:44:19.209 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:24273} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:44:19.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.209+0000 7f22ecc00700 1 -- 172.21.15.27:0/2209390132 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f22cc06e620 msgr2=0x7f22cc070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:19.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.209+0000 7f22ecc00700 1 --2- 172.21.15.27:0/2209390132 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f22cc06e620 0x7f22cc070ad0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f22d400b7a0 tx=0x7f22d4009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:19.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.209+0000 7f22ecc00700 1 -- 172.21.15.27:0/2209390132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22e80708f0 msgr2=0x7f22e8114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:19.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.209+0000 7f22ecc00700 1 --2- 172.21.15.27:0/2209390132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22e80708f0 0x7f22e8114040 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f22d800b790 tx=0x7f22d8009220 comp rx=0 tx=0).stop 2024-04-16T05:44:19.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.209+0000 7f22ecc00700 1 -- 172.21.15.27:0/2209390132 shutdown_connections 2024-04-16T05:44:19.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.209+0000 7f22ecc00700 1 --2- 172.21.15.27:0/2209390132 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f22cc06e620 0x7f22cc070ad0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:19.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.209+0000 7f22ecc00700 1 --2- 172.21.15.27:0/2209390132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22e80708f0 0x7f22e8114040 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:19.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.209+0000 7f22ecc00700 1 --2- 172.21.15.27:0/2209390132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22e8071ea0 0x7f22e8114580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:19.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.209+0000 7f22ecc00700 1 -- 172.21.15.27:0/2209390132 >> 172.21.15.27:0/2209390132 conn(0x7f22e80f8530 msgr2=0x7f22e80f9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:19.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.209+0000 7f22ecc00700 1 -- 172.21.15.27:0/2209390132 shutdown_connections 2024-04-16T05:44:19.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.209+0000 7f22ecc00700 1 -- 172.21.15.27:0/2209390132 wait complete. 2024-04-16T05:44:19.211 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2024-04-16T05:44:19.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.362+0000 7f28fde00700 1 -- 172.21.15.27:0/3757171476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f80662b0 msgr2=0x7f28f80ffa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:19.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.362+0000 7f28fde00700 1 --2- 172.21.15.27:0/3757171476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f80662b0 0x7f28f80ffa70 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f28e8004940 tx=0x7f28e8009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:19.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.362+0000 7f28fde00700 1 -- 172.21.15.27:0/3757171476 shutdown_connections 2024-04-16T05:44:19.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.362+0000 7f28fde00700 1 --2- 172.21.15.27:0/3757171476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28f80fffb0 0x7f28f8102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:19.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.362+0000 7f28fde00700 1 --2- 172.21.15.27:0/3757171476 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f80662b0 0x7f28f80ffa70 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:19.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.362+0000 7f28fde00700 1 -- 172.21.15.27:0/3757171476 >> 172.21.15.27:0/3757171476 conn(0x7f28f80f74c0 msgr2=0x7f28f80f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:19.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.362+0000 7f28fde00700 1 -- 172.21.15.27:0/3757171476 shutdown_connections 2024-04-16T05:44:19.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.362+0000 7f28fde00700 1 -- 172.21.15.27:0/3757171476 wait complete. 2024-04-16T05:44:19.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.363+0000 7f28fde00700 1 Processor -- start 2024-04-16T05:44:19.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.363+0000 7f28fde00700 1 -- start start 2024-04-16T05:44:19.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.363+0000 7f28fde00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28f80662b0 0x7f28f810fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:19.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.363+0000 7f28fde00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f80fffb0 0x7f28f8110260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:19.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.363+0000 7f28fde00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f8110860 con 0x7f28f80fffb0 2024-04-16T05:44:19.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.363+0000 7f28fde00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f8110fa0 con 0x7f28f80662b0 2024-04-16T05:44:19.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.364+0000 7f28f6000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f80fffb0 0x7f28f8110260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:19.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.364+0000 7f28f6000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f80fffb0 0x7f28f8110260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55512/0 (socket says 172.21.15.27:55512) 2024-04-16T05:44:19.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.364+0000 7f28f6000700 1 -- 172.21.15.27:0/2429526898 learned_addr learned my addr 172.21.15.27:0/2429526898 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:44:19.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.364+0000 7f28f6a00700 1 --2- 172.21.15.27:0/2429526898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28f80662b0 0x7f28f810fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:19.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.364+0000 7f28f6000700 1 -- 172.21.15.27:0/2429526898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28f80662b0 msgr2=0x7f28f810fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:19.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.364+0000 7f28f6000700 1 --2- 172.21.15.27:0/2429526898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28f80662b0 0x7f28f810fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:19.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.364+0000 7f28f6000700 1 -- 172.21.15.27:0/2429526898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e8005040 con 0x7f28f80fffb0 2024-04-16T05:44:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.365+0000 7f28f6000700 1 --2- 172.21.15.27:0/2429526898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f80fffb0 0x7f28f8110260 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f28e400c830 tx=0x7f28e4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.365+0000 7f28eea00700 1 -- 172.21.15.27:0/2429526898 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28e400a760 con 0x7f28f80fffb0 2024-04-16T05:44:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.365+0000 7f28eea00700 1 -- 172.21.15.27:0/2429526898 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f28e400ad90 con 0x7f28f80fffb0 2024-04-16T05:44:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.365+0000 7f28eea00700 1 -- 172.21.15.27:0/2429526898 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28e40113a0 con 0x7f28f80fffb0 2024-04-16T05:44:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.365+0000 7f28fde00700 1 -- 172.21.15.27:0/2429526898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f8111280 con 0x7f28f80fffb0 2024-04-16T05:44:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.365+0000 7f28fde00700 1 -- 172.21.15.27:0/2429526898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f8111840 con 0x7f28f80fffb0 2024-04-16T05:44:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.366+0000 7f28fde00700 1 -- 172.21.15.27:0/2429526898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f80f8ff0 con 0x7f28f80fffb0 2024-04-16T05:44:19.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.367+0000 7f28eea00700 1 -- 172.21.15.27:0/2429526898 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 92571+0+0 (secure 0 0 0) 0x7f28e4012400 con 0x7f28f80fffb0 2024-04-16T05:44:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.369+0000 7f28eea00700 1 --2- 172.21.15.27:0/2429526898 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f28dc06e620 0x7f28dc070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.369+0000 7f28eea00700 1 -- 172.21.15.27:0/2429526898 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f28e408df40 con 0x7f28f80fffb0 2024-04-16T05:44:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.370+0000 7f28f6a00700 1 --2- 172.21.15.27:0/2429526898 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f28dc06e620 0x7f28dc070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.371+0000 7f28f6a00700 1 --2- 172.21.15.27:0/2429526898 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f28dc06e620 0x7f28dc070ad0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009fd0 tx=0x7f28e8009e30 comp rx=0 tx=0).ready entity=mgr.24369 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:19.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.374+0000 7f28eea00700 1 -- 172.21.15.27:0/2429526898 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f28e405b140 con 0x7f28f80fffb0 2024-04-16T05:44:19.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.614+0000 7f28fde00700 1 -- 172.21.15.27:0/2429526898 --> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f28f8111b10 con 0x7f28dc06e620 2024-04-16T05:44:19.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.618+0000 7f28eea00700 1 -- 172.21.15.27:0/2429526898 <== mgr.24369 v2:172.21.15.122:6828/871482489 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f28f8111b10 con 0x7f28dc06e620 2024-04-16T05:44:19.618 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:44:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T05:44:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:44:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:44:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [ 2024-04-16T05:44:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr" 2024-04-16T05:44:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-16T05:44:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "2/21 daemons upgraded", 2024-04-16T05:44:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Currently upgrading mgr daemons", 2024-04-16T05:44:19.618 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:44:19.619 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:44:19.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.621+0000 7f28fde00700 1 -- 172.21.15.27:0/2429526898 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f28dc06e620 msgr2=0x7f28dc070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:19.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.621+0000 7f28fde00700 1 --2- 172.21.15.27:0/2429526898 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f28dc06e620 0x7f28dc070ad0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009fd0 tx=0x7f28e8009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:19.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.621+0000 7f28fde00700 1 -- 172.21.15.27:0/2429526898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f80fffb0 msgr2=0x7f28f8110260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:19.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.621+0000 7f28fde00700 1 --2- 172.21.15.27:0/2429526898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f80fffb0 0x7f28f8110260 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f28e400c830 tx=0x7f28e4004470 comp rx=0 tx=0).stop 2024-04-16T05:44:19.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.621+0000 7f28fde00700 1 -- 172.21.15.27:0/2429526898 shutdown_connections 2024-04-16T05:44:19.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.621+0000 7f28fde00700 1 --2- 172.21.15.27:0/2429526898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28f80662b0 0x7f28f810fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:19.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.621+0000 7f28fde00700 1 --2- 172.21.15.27:0/2429526898 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f28dc06e620 0x7f28dc070ad0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:19.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.621+0000 7f28fde00700 1 --2- 172.21.15.27:0/2429526898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f80fffb0 0x7f28f8110260 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:19.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.621+0000 7f28fde00700 1 -- 172.21.15.27:0/2429526898 >> 172.21.15.27:0/2429526898 conn(0x7f28f80f74c0 msgr2=0x7f28f80fbbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:19.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.621+0000 7f28fde00700 1 -- 172.21.15.27:0/2429526898 shutdown_connections 2024-04-16T05:44:19.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.621+0000 7f28fde00700 1 -- 172.21.15.27:0/2429526898 wait complete. 2024-04-16T05:44:19.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.777+0000 7f9d36c00700 1 -- 172.21.15.27:0/3724840874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d300fdee0 msgr2=0x7f9d300fe350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:19.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.777+0000 7f9d36c00700 1 --2- 172.21.15.27:0/3724840874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d300fdee0 0x7f9d300fe350 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c004940 tx=0x7f9d1c009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:19.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.777+0000 7f9d36c00700 1 -- 172.21.15.27:0/3724840874 shutdown_connections 2024-04-16T05:44:19.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.777+0000 7f9d36c00700 1 --2- 172.21.15.27:0/3724840874 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d300fdee0 0x7f9d300fe350 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:19.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.777+0000 7f9d36c00700 1 --2- 172.21.15.27:0/3724840874 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d300fcc30 0x7f9d300fd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:19.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.777+0000 7f9d36c00700 1 -- 172.21.15.27:0/3724840874 >> 172.21.15.27:0/3724840874 conn(0x7f9d300f83d0 msgr2=0x7f9d300fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:19.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.777+0000 7f9d36c00700 1 -- 172.21.15.27:0/3724840874 shutdown_connections 2024-04-16T05:44:19.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.777+0000 7f9d36c00700 1 -- 172.21.15.27:0/3724840874 wait complete. 2024-04-16T05:44:19.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.778+0000 7f9d36c00700 1 Processor -- start 2024-04-16T05:44:19.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.778+0000 7f9d36c00700 1 -- start start 2024-04-16T05:44:19.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.778+0000 7f9d36c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d300fcc30 0x7f9d3010d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:19.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.778+0000 7f9d36c00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d300fdee0 0x7f9d3010ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:19.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.778+0000 7f9d36c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3010e3d0 con 0x7f9d300fcc30 2024-04-16T05:44:19.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.778+0000 7f9d36c00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3010ec20 con 0x7f9d300fdee0 2024-04-16T05:44:19.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.779+0000 7f9d35800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d300fcc30 0x7f9d3010d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:19.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.779+0000 7f9d35800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d300fcc30 0x7f9d3010d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55522/0 (socket says 172.21.15.27:55522) 2024-04-16T05:44:19.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.779+0000 7f9d35800700 1 -- 172.21.15.27:0/3032978393 learned_addr learned my addr 172.21.15.27:0/3032978393 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:44:19.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.779+0000 7f9d34e00700 1 --2- 172.21.15.27:0/3032978393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d300fdee0 0x7f9d3010ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:19.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.779+0000 7f9d35800700 1 -- 172.21.15.27:0/3032978393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d300fdee0 msgr2=0x7f9d3010ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:19.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.779+0000 7f9d35800700 1 --2- 172.21.15.27:0/3032978393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d300fdee0 0x7f9d3010ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:19.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.779+0000 7f9d35800700 1 -- 172.21.15.27:0/3032978393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d1c005040 con 0x7f9d300fcc30 2024-04-16T05:44:19.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.779+0000 7f9d34e00700 1 --2- 172.21.15.27:0/3032978393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d300fdee0 0x7f9d3010ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:44:19.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.780+0000 7f9d35800700 1 --2- 172.21.15.27:0/3032978393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d300fcc30 0x7f9d3010d890 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9d24002a50 tx=0x7f9d240046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:19.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.781+0000 7f9d2d600700 1 -- 172.21.15.27:0/3032978393 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d24007cc0 con 0x7f9d300fcc30 2024-04-16T05:44:19.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.781+0000 7f9d36c00700 1 -- 172.21.15.27:0/3032978393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d3010ef00 con 0x7f9d300fcc30 2024-04-16T05:44:19.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.781+0000 7f9d2d600700 1 -- 172.21.15.27:0/3032978393 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d2400f040 con 0x7f9d300fcc30 2024-04-16T05:44:19.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.781+0000 7f9d2d600700 1 -- 172.21.15.27:0/3032978393 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d24009640 con 0x7f9d300fcc30 2024-04-16T05:44:19.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.781+0000 7f9d36c00700 1 -- 172.21.15.27:0/3032978393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d3010f440 con 0x7f9d300fcc30 2024-04-16T05:44:19.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.782+0000 7f9d36c00700 1 -- 172.21.15.27:0/3032978393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3005f5f0 con 0x7f9d300fcc30 2024-04-16T05:44:19.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.784+0000 7f9d2d600700 1 -- 172.21.15.27:0/3032978393 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 26) v1 ==== 92571+0+0 (secure 0 0 0) 0x7f9d240097a0 con 0x7f9d300fcc30 2024-04-16T05:44:19.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.785+0000 7f9d2d600700 1 --2- 172.21.15.27:0/3032978393 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f9d1806e620 0x7f9d18070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:19.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.785+0000 7f9d2d600700 1 -- 172.21.15.27:0/3032978393 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5808+0+0 (secure 0 0 0) 0x7f9d2408dec0 con 0x7f9d300fcc30 2024-04-16T05:44:19.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.786+0000 7f9d34e00700 1 --2- 172.21.15.27:0/3032978393 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f9d1806e620 0x7f9d18070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:19.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.787+0000 7f9d34e00700 1 --2- 172.21.15.27:0/3032978393 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f9d1806e620 0x7f9d18070ad0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c00b7a0 tx=0x7f9d1c009e30 comp rx=0 tx=0).ready entity=mgr.24369 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:19.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:19.790+0000 7f9d2d600700 1 -- 172.21.15.27:0/3032978393 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f9d2405aeb0 con 0x7f9d300fcc30 2024-04-16T05:44:19.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:19 smithi027 ceph-mon[29206]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:44:19.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:19 smithi027 ceph-mon[29206]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:44:19.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:19 smithi027 ceph-mon[29206]: mgrmap e26: smithi122.sgokkq(active, since 77s), standbys: smithi027.qgzypi 2024-04-16T05:44:19.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:19 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/3750794336' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:44:19.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:19 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/2209390132' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:44:20.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:19 smithi122 ceph-mon[35998]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:44:20.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:19 smithi122 ceph-mon[35998]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:44:20.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:19 smithi122 ceph-mon[35998]: mgrmap e26: smithi122.sgokkq(active, since 77s), standbys: smithi027.qgzypi 2024-04-16T05:44:20.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:19 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/3750794336' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:44:20.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:19 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/2209390132' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:44:20.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:20.108+0000 7f9d36c00700 1 -- 172.21.15.27:0/3032978393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f9d3004bed0 con 0x7f9d300fcc30 2024-04-16T05:44:20.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:20.109+0000 7f9d2d600700 1 -- 172.21.15.27:0/3032978393 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f9d2405aa40 con 0x7f9d300fcc30 2024-04-16T05:44:20.108 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:44:20.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:20.111+0000 7f9d36c00700 1 -- 172.21.15.27:0/3032978393 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f9d1806e620 msgr2=0x7f9d18070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:20.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:20.111+0000 7f9d36c00700 1 --2- 172.21.15.27:0/3032978393 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f9d1806e620 0x7f9d18070ad0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c00b7a0 tx=0x7f9d1c009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:20.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:20.111+0000 7f9d36c00700 1 -- 172.21.15.27:0/3032978393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d300fcc30 msgr2=0x7f9d3010d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:20.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:20.112+0000 7f9d36c00700 1 --2- 172.21.15.27:0/3032978393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d300fcc30 0x7f9d3010d890 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9d24002a50 tx=0x7f9d240046b0 comp rx=0 tx=0).stop 2024-04-16T05:44:20.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:20.112+0000 7f9d36c00700 1 -- 172.21.15.27:0/3032978393 shutdown_connections 2024-04-16T05:44:20.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:20.112+0000 7f9d36c00700 1 --2- 172.21.15.27:0/3032978393 >> [v2:172.21.15.122:6828/871482489,v1:172.21.15.122:6829/871482489] conn(0x7f9d1806e620 0x7f9d18070ad0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:20.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:20.112+0000 7f9d36c00700 1 --2- 172.21.15.27:0/3032978393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d300fcc30 0x7f9d3010d890 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:20.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:20.112+0000 7f9d36c00700 1 --2- 172.21.15.27:0/3032978393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d300fdee0 0x7f9d3010ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:20.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:20.112+0000 7f9d36c00700 1 -- 172.21.15.27:0/3032978393 >> 172.21.15.27:0/3032978393 conn(0x7f9d300f83d0 msgr2=0x7f9d300f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:20.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:20.112+0000 7f9d36c00700 1 -- 172.21.15.27:0/3032978393 shutdown_connections 2024-04-16T05:44:20.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:20.112+0000 7f9d36c00700 1 -- 172.21.15.27:0/3032978393 wait complete. 2024-04-16T05:44:20.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:20 smithi027 ceph-mon[29206]: pgmap v42: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:20.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:20 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/3032978393' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:44:20.924 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:20 smithi122 ceph-mon[35998]: pgmap v42: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:20.924 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:20 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/3032978393' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:44:21.598 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 12, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84320 2024-04-16T05:44:21.600 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 12, success, outsize: 120 2024-04-16T05:44:21.628 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 14, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84412 2024-04-16T05:44:21.629 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 14, success, outsize: 120 2024-04-16T05:44:21.660 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 16, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84507 2024-04-16T05:44:21.661 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 16, success, outsize: 120 2024-04-16T05:44:21.695 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 18, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84608 2024-04-16T05:44:21.695 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 18, success, outsize: 120 2024-04-16T05:44:21.725 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 20, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84706 2024-04-16T05:44:21.725 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 20, success, outsize: 120 2024-04-16T05:44:21.753 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 22, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84798 2024-04-16T05:44:21.753 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 22, success, outsize: 120 2024-04-16T05:44:21.782 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 24, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84893 2024-04-16T05:44:21.782 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 24, success, outsize: 120 2024-04-16T05:44:21.811 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 26, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84994 2024-04-16T05:44:21.811 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 26, success, outsize: 120 2024-04-16T05:44:21.837 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 28, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85092 2024-04-16T05:44:21.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 28, success, outsize: 120 2024-04-16T05:44:21.861 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 30, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85184 2024-04-16T05:44:21.862 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 30, success, outsize: 120 2024-04-16T05:44:21.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 32, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85279 2024-04-16T05:44:21.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 32, success, outsize: 120 2024-04-16T05:44:21.918 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 34, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85380 2024-04-16T05:44:21.918 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 34, success, outsize: 120 2024-04-16T05:44:21.948 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 36, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85478 2024-04-16T05:44:21.948 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 36, success, outsize: 120 2024-04-16T05:44:21.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 38, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85570 2024-04-16T05:44:21.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 38, success, outsize: 120 2024-04-16T05:44:22.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 40, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85665 2024-04-16T05:44:22.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 40, success, outsize: 120 2024-04-16T05:44:22.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:21 smithi122 ceph-mon[35998]: from='client.14740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:44:22.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:21 smithi122 ceph-mon[35998]: pgmap v43: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:22.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 42, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85766 2024-04-16T05:44:22.031 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 42, success, outsize: 120 2024-04-16T05:44:22.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:21 smithi027 ceph-mon[29206]: from='client.14740 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:44:22.060 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:21 smithi027 ceph-mon[29206]: pgmap v43: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:22.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 44, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85864 2024-04-16T05:44:22.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 44, success, outsize: 120 2024-04-16T05:44:22.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 46, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 85956 2024-04-16T05:44:22.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 46, success, outsize: 120 2024-04-16T05:44:22.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 48, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86051 2024-04-16T05:44:22.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 48, success, outsize: 120 2024-04-16T05:44:22.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 50, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86152 2024-04-16T05:44:22.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 50, success, outsize: 120 2024-04-16T05:44:22.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 52, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86250 2024-04-16T05:44:22.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 52, success, outsize: 120 2024-04-16T05:44:22.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 54, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86342 2024-04-16T05:44:22.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 54, success, outsize: 120 2024-04-16T05:44:22.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 56, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86437 2024-04-16T05:44:22.222 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 56, success, outsize: 120 2024-04-16T05:44:22.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 58, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86538 2024-04-16T05:44:22.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 58, success, outsize: 120 2024-04-16T05:44:22.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 60, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86636 2024-04-16T05:44:22.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 60, success, outsize: 120 2024-04-16T05:44:22.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 62, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86728 2024-04-16T05:44:22.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 62, success, outsize: 120 2024-04-16T05:44:22.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 64, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86823 2024-04-16T05:44:22.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 64, success, outsize: 120 2024-04-16T05:44:22.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 66, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86924 2024-04-16T05:44:22.344 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 66, success, outsize: 120 2024-04-16T05:44:24.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: pgmap v44: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:24.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:24.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:24.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:44:24.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:44:24.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:24.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:44:24.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mgr fail", "who": "smithi122.sgokkq"}]: dispatch 2024-04-16T05:44:24.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: Activating manager daemon smithi027.qgzypi 2024-04-16T05:44:24.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: osdmap e48: 6 total, 6 up, 6 in 2024-04-16T05:44:24.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd='[{"prefix": "mgr fail", "who": "smithi122.sgokkq"}]': finished 2024-04-16T05:44:24.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: mgrmap e27: smithi027.qgzypi(active, starting, since 0.0175163s) 2024-04-16T05:44:24.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-16T05:44:24.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:44:24.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.ejoytk"}]: dispatch 2024-04-16T05:44:24.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.hkhpuq"}]: dispatch 2024-04-16T05:44:24.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.lojjdp"}]: dispatch 2024-04-16T05:44:24.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.rulewi"}]: dispatch 2024-04-16T05:44:24.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr metadata", "who": "smithi027.qgzypi", "id": "smithi027.qgzypi"}]: dispatch 2024-04-16T05:44:24.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T05:44:24.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T05:44:24.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T05:44:24.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T05:44:24.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T05:44:24.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T05:44:24.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T05:44:24.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T05:44:24.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T05:44:24.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: Manager daemon smithi027.qgzypi is now available 2024-04-16T05:44:24.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:44:24.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:44:24.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.qgzypi/mirror_snapshot_schedule"}]: dispatch 2024-04-16T05:44:24.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:24 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.qgzypi/trash_purge_schedule"}]: dispatch 2024-04-16T05:44:24.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: pgmap v44: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:24.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:24.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:24.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:44:24.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:44:24.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' 2024-04-16T05:44:24.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:44:24.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "mgr fail", "who": "smithi122.sgokkq"}]: dispatch 2024-04-16T05:44:24.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: Activating manager daemon smithi027.qgzypi 2024-04-16T05:44:24.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: osdmap e48: 6 total, 6 up, 6 in 2024-04-16T05:44:24.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.24369 172.21.15.122:0/2565807960' entity='mgr.smithi122.sgokkq' cmd='[{"prefix": "mgr fail", "who": "smithi122.sgokkq"}]': finished 2024-04-16T05:44:24.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: mgrmap e27: smithi027.qgzypi(active, starting, since 0.0175163s) 2024-04-16T05:44:24.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-16T05:44:24.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:44:24.524 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.ejoytk"}]: dispatch 2024-04-16T05:44:24.524 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.hkhpuq"}]: dispatch 2024-04-16T05:44:24.524 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.lojjdp"}]: dispatch 2024-04-16T05:44:24.524 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.rulewi"}]: dispatch 2024-04-16T05:44:24.524 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr metadata", "who": "smithi027.qgzypi", "id": "smithi027.qgzypi"}]: dispatch 2024-04-16T05:44:24.524 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T05:44:24.524 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T05:44:24.524 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T05:44:24.524 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T05:44:24.524 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T05:44:24.524 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T05:44:24.525 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T05:44:24.525 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T05:44:24.525 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T05:44:24.525 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: Manager daemon smithi027.qgzypi is now available 2024-04-16T05:44:24.525 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:44:24.525 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:44:24.525 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.qgzypi/mirror_snapshot_schedule"}]: dispatch 2024-04-16T05:44:24.525 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:24 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.qgzypi/trash_purge_schedule"}]: dispatch 2024-04-16T05:44:25.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:25 smithi027 ceph-mon[29206]: [16/Apr/2024:05:44:24] ENGINE Bus STARTING 2024-04-16T05:44:25.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:25 smithi027 ceph-mon[29206]: [16/Apr/2024:05:44:24] ENGINE Serving on http://172.21.15.27:8765 2024-04-16T05:44:25.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:25 smithi027 ceph-mon[29206]: [16/Apr/2024:05:44:24] ENGINE Serving on https://172.21.15.27:7150 2024-04-16T05:44:25.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:25 smithi027 ceph-mon[29206]: [16/Apr/2024:05:44:24] ENGINE Bus STARTED 2024-04-16T05:44:25.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:25 smithi027 ceph-mon[29206]: mgrmap e28: smithi027.qgzypi(active, since 1.01818s) 2024-04-16T05:44:25.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:25 smithi027 ceph-mon[29206]: pgmap v3: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:25.961 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:25 smithi122 ceph-mon[35998]: [16/Apr/2024:05:44:24] ENGINE Bus STARTING 2024-04-16T05:44:25.962 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:25 smithi122 ceph-mon[35998]: [16/Apr/2024:05:44:24] ENGINE Serving on http://172.21.15.27:8765 2024-04-16T05:44:25.962 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:25 smithi122 ceph-mon[35998]: [16/Apr/2024:05:44:24] ENGINE Serving on https://172.21.15.27:7150 2024-04-16T05:44:25.962 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:25 smithi122 ceph-mon[35998]: [16/Apr/2024:05:44:24] ENGINE Bus STARTED 2024-04-16T05:44:25.962 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:25 smithi122 ceph-mon[35998]: mgrmap e28: smithi027.qgzypi(active, since 1.01818s) 2024-04-16T05:44:25.962 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:25 smithi122 ceph-mon[35998]: pgmap v3: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:26.826 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:26 smithi122 ceph-mon[35998]: pgmap v4: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:27.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:26 smithi027 ceph-mon[29206]: pgmap v4: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:28.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:27 smithi122 ceph-mon[35998]: mgrmap e29: smithi027.qgzypi(active, since 3s) 2024-04-16T05:44:28.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:27 smithi027 ceph-mon[29206]: mgrmap e29: smithi027.qgzypi(active, since 3s) 2024-04-16T05:44:29.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:28 smithi122 ceph-mon[35998]: pgmap v5: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:29.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:28 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:29.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:28 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:29.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:28 smithi027 ceph-mon[29206]: pgmap v5: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:29.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:28 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:29.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:28 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:30.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:29 smithi122 ceph-mon[35998]: mgrmap e30: smithi027.qgzypi(active, since 5s) 2024-04-16T05:44:30.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:29 smithi027 ceph-mon[29206]: mgrmap e30: smithi027.qgzypi(active, since 5s) 2024-04-16T05:44:31.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:30 smithi122 ceph-mon[35998]: pgmap v6: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:31.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:30 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:31.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:30 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:31.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:30 smithi027 ceph-mon[29206]: pgmap v6: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:31.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:30 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:31.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:30 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:32.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:32 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:32.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:32 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:32.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:32 smithi122 ceph-mon[35998]: pgmap v7: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-16T05:44:32.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:32 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:32.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:32 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:32.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:32 smithi027 ceph-mon[29206]: pgmap v7: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-16T05:44:33.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:33 smithi122 ceph-mon[35998]: Standby manager daemon smithi122.sgokkq started 2024-04-16T05:44:33.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:33 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.122:0/1656577371' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.sgokkq/crt"}]: dispatch 2024-04-16T05:44:33.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:33 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.122:0/1656577371' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T05:44:33.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:33 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.122:0/1656577371' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.sgokkq/key"}]: dispatch 2024-04-16T05:44:33.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:33 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.122:0/1656577371' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T05:44:33.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:33 smithi027 ceph-mon[29206]: Standby manager daemon smithi122.sgokkq started 2024-04-16T05:44:33.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:33 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.122:0/1656577371' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.sgokkq/crt"}]: dispatch 2024-04-16T05:44:33.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:33 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.122:0/1656577371' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T05:44:33.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:33 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.122:0/1656577371' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.sgokkq/key"}]: dispatch 2024-04-16T05:44:33.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:33 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.122:0/1656577371' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T05:44:34.706 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:34 smithi027 ceph-mon[29206]: mgrmap e31: smithi027.qgzypi(active, since 9s), standbys: smithi122.sgokkq 2024-04-16T05:44:34.706 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:34 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr metadata", "who": "smithi122.sgokkq", "id": "smithi122.sgokkq"}]: dispatch 2024-04-16T05:44:34.706 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:34 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:34.706 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:34 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:34.706 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:34 smithi027 ceph-mon[29206]: pgmap v8: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-16T05:44:34.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:34 smithi122 ceph-mon[35998]: mgrmap e31: smithi027.qgzypi(active, since 9s), standbys: smithi122.sgokkq 2024-04-16T05:44:34.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:34 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr metadata", "who": "smithi122.sgokkq", "id": "smithi122.sgokkq"}]: dispatch 2024-04-16T05:44:34.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:34 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:34.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:34 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:34.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:34 smithi122 ceph-mon[35998]: pgmap v8: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-16T05:44:37.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:36 smithi122 ceph-mon[35998]: pgmap v9: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-16T05:44:37.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:36 smithi027 ceph-mon[29206]: pgmap v9: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-16T05:44:38.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 68, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88309 2024-04-16T05:44:38.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 68, success, outsize: 120 2024-04-16T05:44:38.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 70, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88407 2024-04-16T05:44:38.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 70, success, outsize: 120 2024-04-16T05:44:38.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 72, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88508 2024-04-16T05:44:38.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 72, success, outsize: 120 2024-04-16T05:44:38.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 74, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88615 2024-04-16T05:44:38.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 74, success, outsize: 120 2024-04-16T05:44:38.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 76, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88719 2024-04-16T05:44:38.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 76, success, outsize: 120 2024-04-16T05:44:38.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 78, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88817 2024-04-16T05:44:38.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 78, success, outsize: 120 2024-04-16T05:44:38.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 80, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88918 2024-04-16T05:44:38.171 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 80, success, outsize: 120 2024-04-16T05:44:38.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 82, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89025 2024-04-16T05:44:38.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 82, success, outsize: 120 2024-04-16T05:44:38.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 84, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89131 2024-04-16T05:44:38.220 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 84, success, outsize: 120 2024-04-16T05:44:38.240 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 86, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89230 2024-04-16T05:44:38.240 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 86, success, outsize: 120 2024-04-16T05:44:38.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 88, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89331 2024-04-16T05:44:38.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 88, success, outsize: 120 2024-04-16T05:44:38.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 90, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89438 2024-04-16T05:44:38.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 90, success, outsize: 120 2024-04-16T05:44:38.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 92, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89542 2024-04-16T05:44:38.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 92, success, outsize: 120 2024-04-16T05:44:38.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 94, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89640 2024-04-16T05:44:38.334 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 94, success, outsize: 120 2024-04-16T05:44:38.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 96, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89741 2024-04-16T05:44:38.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 96, success, outsize: 120 2024-04-16T05:44:38.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 98, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89848 2024-04-16T05:44:38.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 98, success, outsize: 120 2024-04-16T05:44:38.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 100, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89953 2024-04-16T05:44:38.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 100, success, outsize: 120 2024-04-16T05:44:38.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 102, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 90055 2024-04-16T05:44:38.424 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 102, success, outsize: 120 2024-04-16T05:44:38.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 104, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 90156 2024-04-16T05:44:38.443 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 104, success, outsize: 120 2024-04-16T05:44:38.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 106, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 90263 2024-04-16T05:44:38.462 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 106, success, outsize: 120 2024-04-16T05:44:38.480 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 108, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 90367 2024-04-16T05:44:38.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 108, success, outsize: 120 2024-04-16T05:44:38.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 110, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 90465 2024-04-16T05:44:38.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 110, success, outsize: 120 2024-04-16T05:44:38.519 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 112, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 90566 2024-04-16T05:44:38.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 112, success, outsize: 120 2024-04-16T05:44:38.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 114, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 90673 2024-04-16T05:44:38.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 114, success, outsize: 120 2024-04-16T05:44:38.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 116, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 90777 2024-04-16T05:44:38.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 116, success, outsize: 120 2024-04-16T05:44:38.585 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 118, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 90875 2024-04-16T05:44:38.585 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 118, success, outsize: 120 2024-04-16T05:44:38.605 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 120, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 90976 2024-04-16T05:44:38.605 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 120, success, outsize: 120 2024-04-16T05:44:38.627 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 122, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 91083 2024-04-16T05:44:38.627 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 122, success, outsize: 120 2024-04-16T05:44:39.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:38 smithi122 ceph-mon[35998]: pgmap v10: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T05:44:39.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:38 smithi027 ceph-mon[29206]: pgmap v10: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T05:44:40.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:39 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:44:40.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:39 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:40.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:39 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:40.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:39 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:44:40.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:39 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:44:40.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:39 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:44:40.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:39 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:40.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:39 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:40.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:39 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:44:40.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:39 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:44:41.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:40 smithi122 ceph-mon[35998]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-16T05:44:41.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:40 smithi122 ceph-mon[35998]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-16T05:44:41.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:40 smithi122 ceph-mon[35998]: pgmap v11: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T05:44:41.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:40 smithi027 ceph-mon[29206]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-16T05:44:41.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:40 smithi027 ceph-mon[29206]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-16T05:44:41.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:40 smithi027 ceph-mon[29206]: pgmap v11: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T05:44:42.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:41 smithi122 ceph-mon[35998]: Updating smithi122:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:44:42.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:41 smithi122 ceph-mon[35998]: Updating smithi027:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:44:42.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:41 smithi027 ceph-mon[29206]: Updating smithi122:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:44:42.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:41 smithi027 ceph-mon[29206]: Updating smithi027:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:44:43.038 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:42 smithi027 ceph-mon[29206]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-16T05:44:43.038 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:42 smithi027 ceph-mon[29206]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-16T05:44:43.038 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:42 smithi027 ceph-mon[29206]: pgmap v12: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T05:44:43.039 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:42 smithi027 ceph-mon[29206]: Updating smithi122:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T05:44:43.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:42 smithi122 ceph-mon[35998]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-16T05:44:43.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:42 smithi122 ceph-mon[35998]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-16T05:44:43.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:42 smithi122 ceph-mon[35998]: pgmap v12: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T05:44:43.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:42 smithi122 ceph-mon[35998]: Updating smithi122:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T05:44:44.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:43 smithi027 ceph-mon[29206]: Updating smithi027:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T05:44:44.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:43 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:44.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:43 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:44.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:43 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:44.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:43 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:44.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:43 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:44.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:43 smithi122 ceph-mon[35998]: Updating smithi027:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T05:44:44.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:43 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:44.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:43 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:44.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:43 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:44.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:43 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:44.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:43 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:45.270 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:44 smithi027 ceph-mon[29206]: Reconfiguring prometheus.smithi027 (dependencies changed)... 2024-04-16T05:44:45.270 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:44 smithi027 ceph-mon[29206]: Reconfiguring daemon prometheus.smithi027 on smithi027 2024-04-16T05:44:45.270 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:44 smithi027 ceph-mon[29206]: pgmap v13: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:45.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:44 smithi122 ceph-mon[35998]: Reconfiguring prometheus.smithi027 (dependencies changed)... 2024-04-16T05:44:45.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:44 smithi122 ceph-mon[35998]: Reconfiguring daemon prometheus.smithi027 on smithi027 2024-04-16T05:44:45.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:44 smithi122 ceph-mon[35998]: pgmap v13: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:47.077 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:46 smithi027 ceph-mon[29206]: pgmap v14: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:47.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:46 smithi122 ceph-mon[35998]: pgmap v14: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:49.202 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:48 smithi027 ceph-mon[29206]: pgmap v15: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:49.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:48 smithi122 ceph-mon[35998]: pgmap v15: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:50.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.279+0000 7f9295200700 1 -- 172.21.15.27:0/1699463773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9290103420 msgr2=0x7f9290103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:50.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.279+0000 7f9295200700 1 --2- 172.21.15.27:0/1699463773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9290103420 0x7f9290103830 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9280004940 tx=0x7f9280009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:50.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.279+0000 7f9295200700 1 -- 172.21.15.27:0/1699463773 shutdown_connections 2024-04-16T05:44:50.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.279+0000 7f9295200700 1 --2- 172.21.15.27:0/1699463773 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92901046d0 0x7f9290104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:50.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.279+0000 7f9295200700 1 --2- 172.21.15.27:0/1699463773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9290103420 0x7f9290103830 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:50.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.279+0000 7f9295200700 1 -- 172.21.15.27:0/1699463773 >> 172.21.15.27:0/1699463773 conn(0x7f9290072640 msgr2=0x7f9290074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.279+0000 7f9295200700 1 -- 172.21.15.27:0/1699463773 shutdown_connections 2024-04-16T05:44:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.279+0000 7f9295200700 1 -- 172.21.15.27:0/1699463773 wait complete. 2024-04-16T05:44:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.280+0000 7f9295200700 1 Processor -- start 2024-04-16T05:44:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.280+0000 7f9295200700 1 -- start start 2024-04-16T05:44:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.281+0000 7f9295200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9290103420 0x7f9290113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.281+0000 7f9295200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92901046d0 0x7f92901144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.281+0000 7f9295200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9290114aa0 con 0x7f92901046d0 2024-04-16T05:44:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.281+0000 7f9295200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9290114c10 con 0x7f9290103420 2024-04-16T05:44:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.281+0000 7f928d400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92901046d0 0x7f92901144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.281+0000 7f928d400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92901046d0 0x7f92901144a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36016/0 (socket says 172.21.15.27:36016) 2024-04-16T05:44:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.281+0000 7f928d400700 1 -- 172.21.15.27:0/430352701 learned_addr learned my addr 172.21.15.27:0/430352701 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:44:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.281+0000 7f928de00700 1 --2- 172.21.15.27:0/430352701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9290103420 0x7f9290113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:50.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.282+0000 7f928d400700 1 -- 172.21.15.27:0/430352701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9290103420 msgr2=0x7f9290113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:50.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.282+0000 7f928d400700 1 --2- 172.21.15.27:0/430352701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9290103420 0x7f9290113f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:50.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.282+0000 7f928d400700 1 -- 172.21.15.27:0/430352701 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9280005040 con 0x7f92901046d0 2024-04-16T05:44:50.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.282+0000 7f928d400700 1 --2- 172.21.15.27:0/430352701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92901046d0 0x7f92901144a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f927800c8c0 tx=0x7f92780044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:50.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.282+0000 7f9286000700 1 -- 172.21.15.27:0/430352701 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f927800a980 con 0x7f92901046d0 2024-04-16T05:44:50.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.282+0000 7f9295200700 1 -- 172.21.15.27:0/430352701 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9290119670 con 0x7f92901046d0 2024-04-16T05:44:50.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.282+0000 7f9286000700 1 -- 172.21.15.27:0/430352701 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f927800ce60 con 0x7f92901046d0 2024-04-16T05:44:50.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.282+0000 7f9286000700 1 -- 172.21.15.27:0/430352701 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92780115e0 con 0x7f92901046d0 2024-04-16T05:44:50.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.283+0000 7f9295200700 1 -- 172.21.15.27:0/430352701 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9290119c30 con 0x7f92901046d0 2024-04-16T05:44:50.282 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.284+0000 7f9295200700 1 -- 172.21.15.27:0/430352701 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f929004c710 con 0x7f92901046d0 2024-04-16T05:44:50.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.285+0000 7f9286000700 1 -- 172.21.15.27:0/430352701 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 31) v1 ==== 92569+0+0 (secure 0 0 0) 0x7f9278012400 con 0x7f92901046d0 2024-04-16T05:44:50.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.286+0000 7f9286000700 1 --2- 172.21.15.27:0/430352701 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f927406e630 0x7f9274070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:50.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.286+0000 7f9286000700 1 -- 172.21.15.27:0/430352701 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f927808e190 con 0x7f92901046d0 2024-04-16T05:44:50.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.286+0000 7f928de00700 1 --2- 172.21.15.27:0/430352701 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f927406e630 0x7f9274070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:50.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.287+0000 7f928de00700 1 --2- 172.21.15.27:0/430352701 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f927406e630 0x7f9274070ae0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9280004f90 tx=0x7f9280009e30 comp rx=0 tx=0).ready entity=mgr.14716 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:50.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.292+0000 7f9286000700 1 -- 172.21.15.27:0/430352701 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f927805b230 con 0x7f92901046d0 2024-04-16T05:44:50.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.555+0000 7f9295200700 1 -- 172.21.15.27:0/430352701 --> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9290108a60 con 0x7f927406e630 2024-04-16T05:44:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.558+0000 7f9286000700 1 -- 172.21.15.27:0/430352701 <== mgr.14716 v2:172.21.15.27:6800/1090905394 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f9290108a60 con 0x7f927406e630 2024-04-16T05:44:50.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.562+0000 7f9295200700 1 -- 172.21.15.27:0/430352701 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f927406e630 msgr2=0x7f9274070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.562+0000 7f9295200700 1 --2- 172.21.15.27:0/430352701 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f927406e630 0x7f9274070ae0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9280004f90 tx=0x7f9280009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.562+0000 7f9295200700 1 -- 172.21.15.27:0/430352701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92901046d0 msgr2=0x7f92901144a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.562+0000 7f9295200700 1 --2- 172.21.15.27:0/430352701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92901046d0 0x7f92901144a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f927800c8c0 tx=0x7f92780044c0 comp rx=0 tx=0).stop 2024-04-16T05:44:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.562+0000 7f9295200700 1 -- 172.21.15.27:0/430352701 shutdown_connections 2024-04-16T05:44:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.562+0000 7f9295200700 1 --2- 172.21.15.27:0/430352701 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f927406e630 0x7f9274070ae0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.562+0000 7f9295200700 1 --2- 172.21.15.27:0/430352701 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9290103420 0x7f9290113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.562+0000 7f9295200700 1 --2- 172.21.15.27:0/430352701 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92901046d0 0x7f92901144a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.562+0000 7f9295200700 1 -- 172.21.15.27:0/430352701 >> 172.21.15.27:0/430352701 conn(0x7f9290072640 msgr2=0x7f9290073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.562+0000 7f9295200700 1 -- 172.21.15.27:0/430352701 shutdown_connections 2024-04-16T05:44:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.563+0000 7f9295200700 1 -- 172.21.15.27:0/430352701 wait complete. 2024-04-16T05:44:50.577 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:44:50.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.750+0000 7f46c0400700 1 -- 172.21.15.27:0/218228889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b8066b80 msgr2=0x7f46b80ff9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:50.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.750+0000 7f46c0400700 1 --2- 172.21.15.27:0/218228889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b8066b80 0x7f46b80ff9d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f46ac004940 tx=0x7f46ac009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:50.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.750+0000 7f46c0400700 1 -- 172.21.15.27:0/218228889 shutdown_connections 2024-04-16T05:44:50.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.750+0000 7f46c0400700 1 --2- 172.21.15.27:0/218228889 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46b80fff10 0x7f46b8102330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:50.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.750+0000 7f46c0400700 1 --2- 172.21.15.27:0/218228889 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b8066b80 0x7f46b80ff9d0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:50.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.750+0000 7f46c0400700 1 -- 172.21.15.27:0/218228889 >> 172.21.15.27:0/218228889 conn(0x7f46b80f7440 msgr2=0x7f46b80f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:50.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.750+0000 7f46c0400700 1 -- 172.21.15.27:0/218228889 shutdown_connections 2024-04-16T05:44:50.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.750+0000 7f46c0400700 1 -- 172.21.15.27:0/218228889 wait complete. 2024-04-16T05:44:50.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.751+0000 7f46c0400700 1 Processor -- start 2024-04-16T05:44:50.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.751+0000 7f46c0400700 1 -- start start 2024-04-16T05:44:50.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.751+0000 7f46c0400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b8066b80 0x7f46b810fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:50.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.751+0000 7f46c0400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46b80fff10 0x7f46b81102a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:50.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.751+0000 7f46c0400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b8110f60 con 0x7f46b8066b80 2024-04-16T05:44:50.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.751+0000 7f46c0400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b81110d0 con 0x7f46b80fff10 2024-04-16T05:44:50.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.752+0000 7f46bda00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b8066b80 0x7f46b810fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:50.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.752+0000 7f46bda00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b8066b80 0x7f46b810fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36032/0 (socket says 172.21.15.27:36032) 2024-04-16T05:44:50.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.752+0000 7f46bda00700 1 -- 172.21.15.27:0/2350840936 learned_addr learned my addr 172.21.15.27:0/2350840936 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:44:50.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.752+0000 7f46bd000700 1 --2- 172.21.15.27:0/2350840936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46b80fff10 0x7f46b81102a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.752+0000 7f46bda00700 1 -- 172.21.15.27:0/2350840936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46b80fff10 msgr2=0x7f46b81102a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.752+0000 7f46bda00700 1 --2- 172.21.15.27:0/2350840936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46b80fff10 0x7f46b81102a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.752+0000 7f46bda00700 1 -- 172.21.15.27:0/2350840936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ac005040 con 0x7f46b8066b80 2024-04-16T05:44:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.752+0000 7f46bd000700 1 --2- 172.21.15.27:0/2350840936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46b80fff10 0x7f46b81102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:44:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.752+0000 7f46bda00700 1 --2- 172.21.15.27:0/2350840936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b8066b80 0x7f46b810fd60 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f46ac00b970 tx=0x7f46ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.753+0000 7f46b5600700 1 -- 172.21.15.27:0/2350840936 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46ac00e040 con 0x7f46b8066b80 2024-04-16T05:44:50.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.753+0000 7f46b5600700 1 -- 172.21.15.27:0/2350840936 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f46ac004290 con 0x7f46b8066b80 2024-04-16T05:44:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.753+0000 7f46c0400700 1 -- 172.21.15.27:0/2350840936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b8111350 con 0x7f46b8066b80 2024-04-16T05:44:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.753+0000 7f46b5600700 1 -- 172.21.15.27:0/2350840936 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46ac01c630 con 0x7f46b8066b80 2024-04-16T05:44:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.753+0000 7f46c0400700 1 -- 172.21.15.27:0/2350840936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b8111910 con 0x7f46b8066b80 2024-04-16T05:44:50.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.754+0000 7f46c0400700 1 -- 172.21.15.27:0/2350840936 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b805f5f0 con 0x7f46b8066b80 2024-04-16T05:44:50.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.755+0000 7f46b5600700 1 -- 172.21.15.27:0/2350840936 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 31) v1 ==== 92569+0+0 (secure 0 0 0) 0x7f46ac00bbf0 con 0x7f46b8066b80 2024-04-16T05:44:50.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.756+0000 7f46b5600700 1 --2- 172.21.15.27:0/2350840936 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f46a006e620 0x7f46a0070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:50.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.757+0000 7f46b5600700 1 -- 172.21.15.27:0/2350840936 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f46ac08f5e0 con 0x7f46b8066b80 2024-04-16T05:44:50.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.757+0000 7f46bd000700 1 --2- 172.21.15.27:0/2350840936 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f46a006e620 0x7f46a0070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:50.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.757+0000 7f46bd000700 1 --2- 172.21.15.27:0/2350840936 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f46a006e620 0x7f46a0070ad0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f46a4006fd0 tx=0x7f46a4008040 comp rx=0 tx=0).ready entity=mgr.14716 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:50.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:50.762+0000 7f46b5600700 1 -- 172.21.15.27:0/2350840936 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f46ac05c4d0 con 0x7f46b8066b80 2024-04-16T05:44:51.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.016+0000 7f46c0400700 1 -- 172.21.15.27:0/2350840936 --> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f46b8111bf0 con 0x7f46a006e620 2024-04-16T05:44:51.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.018+0000 7f46b5600700 1 -- 172.21.15.27:0/2350840936 <== mgr.14716 v2:172.21.15.27:6800/1090905394 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f46b8111bf0 con 0x7f46a006e620 2024-04-16T05:44:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.021+0000 7f46c0400700 1 -- 172.21.15.27:0/2350840936 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f46a006e620 msgr2=0x7f46a0070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.021+0000 7f46c0400700 1 --2- 172.21.15.27:0/2350840936 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f46a006e620 0x7f46a0070ad0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f46a4006fd0 tx=0x7f46a4008040 comp rx=0 tx=0).stop 2024-04-16T05:44:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.021+0000 7f46c0400700 1 -- 172.21.15.27:0/2350840936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b8066b80 msgr2=0x7f46b810fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:51.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.021+0000 7f46c0400700 1 --2- 172.21.15.27:0/2350840936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b8066b80 0x7f46b810fd60 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f46ac00b970 tx=0x7f46ac009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:51.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.021+0000 7f46c0400700 1 -- 172.21.15.27:0/2350840936 shutdown_connections 2024-04-16T05:44:51.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.021+0000 7f46c0400700 1 --2- 172.21.15.27:0/2350840936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b8066b80 0x7f46b810fd60 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:51.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.021+0000 7f46c0400700 1 --2- 172.21.15.27:0/2350840936 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f46a006e620 0x7f46a0070ad0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:51.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.021+0000 7f46c0400700 1 --2- 172.21.15.27:0/2350840936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46b80fff10 0x7f46b81102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:51.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.022+0000 7f46c0400700 1 -- 172.21.15.27:0/2350840936 >> 172.21.15.27:0/2350840936 conn(0x7f46b80f7440 msgr2=0x7f46b80fba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:51.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.022+0000 7f46c0400700 1 -- 172.21.15.27:0/2350840936 shutdown_connections 2024-04-16T05:44:51.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.022+0000 7f46c0400700 1 -- 172.21.15.27:0/2350840936 wait complete. 2024-04-16T05:44:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.179+0000 7ffb06200700 1 -- 172.21.15.27:0/1476042612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb000708f0 msgr2=0x7ffb00070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.179+0000 7ffb06200700 1 --2- 172.21.15.27:0/1476042612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb000708f0 0x7ffb00070d60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0004940 tx=0x7ffaf0009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.179+0000 7ffb06200700 1 -- 172.21.15.27:0/1476042612 shutdown_connections 2024-04-16T05:44:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.179+0000 7ffb06200700 1 --2- 172.21.15.27:0/1476042612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb000708f0 0x7ffb00070d60 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.179+0000 7ffb06200700 1 --2- 172.21.15.27:0/1476042612 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb00071ea0 0x7ffb000703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.179+0000 7ffb06200700 1 -- 172.21.15.27:0/1476042612 >> 172.21.15.27:0/1476042612 conn(0x7ffb000f83d0 msgr2=0x7ffb000fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.179+0000 7ffb06200700 1 -- 172.21.15.27:0/1476042612 shutdown_connections 2024-04-16T05:44:51.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.179+0000 7ffb06200700 1 -- 172.21.15.27:0/1476042612 wait complete. 2024-04-16T05:44:51.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.180+0000 7ffb06200700 1 Processor -- start 2024-04-16T05:44:51.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.180+0000 7ffb06200700 1 -- start start 2024-04-16T05:44:51.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.181+0000 7ffb06200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb000708f0 0x7ffb00113f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:51.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.181+0000 7ffb06200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb00071ea0 0x7ffb00114450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.181+0000 7ffb06200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb00115160 con 0x7ffb000708f0 2024-04-16T05:44:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.181+0000 7ffb06200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb001152d0 con 0x7ffb00071ea0 2024-04-16T05:44:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.181+0000 7ffb04e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb000708f0 0x7ffb00113f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.181+0000 7ffb04e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb000708f0 0x7ffb00113f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36058/0 (socket says 172.21.15.27:36058) 2024-04-16T05:44:51.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.181+0000 7ffb04e00700 1 -- 172.21.15.27:0/3941456931 learned_addr learned my addr 172.21.15.27:0/3941456931 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:44:51.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.181+0000 7ffaffe00700 1 --2- 172.21.15.27:0/3941456931 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb00071ea0 0x7ffb00114450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:51.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.182+0000 7ffb04e00700 1 -- 172.21.15.27:0/3941456931 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb00071ea0 msgr2=0x7ffb00114450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:51.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.182+0000 7ffb04e00700 1 --2- 172.21.15.27:0/3941456931 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb00071ea0 0x7ffb00114450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:51.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.182+0000 7ffb04e00700 1 -- 172.21.15.27:0/3941456931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf0005040 con 0x7ffb000708f0 2024-04-16T05:44:51.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.182+0000 7ffb04e00700 1 --2- 172.21.15.27:0/3941456931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb000708f0 0x7ffb00113f10 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ffaf400b790 tx=0x7ffaf4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:51.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.182+0000 7ffafcc00700 1 -- 172.21.15.27:0/3941456931 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffaf400f6f0 con 0x7ffb000708f0 2024-04-16T05:44:51.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.182+0000 7ffafcc00700 1 -- 172.21.15.27:0/3941456931 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffaf400fd20 con 0x7ffb000708f0 2024-04-16T05:44:51.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.182+0000 7ffb06200700 1 -- 172.21.15.27:0/3941456931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb001155b0 con 0x7ffb000708f0 2024-04-16T05:44:51.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.182+0000 7ffafcc00700 1 -- 172.21.15.27:0/3941456931 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffaf40103a0 con 0x7ffb000708f0 2024-04-16T05:44:51.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.182+0000 7ffb06200700 1 -- 172.21.15.27:0/3941456931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb00115b70 con 0x7ffb000708f0 2024-04-16T05:44:51.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.184+0000 7ffb06200700 1 -- 172.21.15.27:0/3941456931 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb0010c5e0 con 0x7ffb000708f0 2024-04-16T05:44:51.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.185+0000 7ffafcc00700 1 -- 172.21.15.27:0/3941456931 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 31) v1 ==== 92569+0+0 (secure 0 0 0) 0x7ffaf4011400 con 0x7ffb000708f0 2024-04-16T05:44:51.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.186+0000 7ffafcc00700 1 --2- 172.21.15.27:0/3941456931 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7ffae806e620 0x7ffae8070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:51.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.186+0000 7ffafcc00700 1 -- 172.21.15.27:0/3941456931 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7ffaf408de30 con 0x7ffb000708f0 2024-04-16T05:44:51.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.187+0000 7ffaffe00700 1 --2- 172.21.15.27:0/3941456931 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7ffae806e620 0x7ffae8070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:51.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.187+0000 7ffaffe00700 1 --2- 172.21.15.27:0/3941456931 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7ffae806e620 0x7ffae8070ad0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ffaf000b3b0 tx=0x7ffaf0009e30 comp rx=0 tx=0).ready entity=mgr.14716 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:51.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.191+0000 7ffafcc00700 1 -- 172.21.15.27:0/3941456931 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7ffaf405aeb0 con 0x7ffb000708f0 2024-04-16T05:44:51.192 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:50 smithi027 ceph-mon[29206]: pgmap v16: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:51.192 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:50 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:51.192 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:50 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:51.193 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:50 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T05:44:51.193 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:50 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:44:51.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:50 smithi122 ceph-mon[35998]: pgmap v16: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:51.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:50 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:51.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:50 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:44:51.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:50 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T05:44:51.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:50 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:44:51.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.434+0000 7ffb06200700 1 -- 172.21.15.27:0/3941456931 --> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb0010bdd0 con 0x7ffae806e620 2024-04-16T05:44:51.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.464+0000 7ffafcc00700 1 -- 172.21.15.27:0/3941456931 <== mgr.14716 v2:172.21.15.27:6800/1090905394 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7ffb0010bdd0 con 0x7ffae806e620 2024-04-16T05:44:51.463 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:44:51.463 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (71s) 20s ago 11m 13.4M - 0.25.0 c8568f914cd2 4ed63a8a6b33 2024-04-16T05:44:51.464 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (12m) 20s ago 12m 6665k - 17.2.7-847-g45317f95 b333581ccdf5 f1acb6959c63 2024-04-16T05:44:51.464 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (10m) 23s ago 10m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 e81efc20e8c0 2024-04-16T05:44:51.464 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (64s) 20s ago 11m 74.9M - 9.4.7 954c08fa6188 f7ee2a1afd12 2024-04-16T05:44:51.464 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (4m) 20s ago 4m 23.2M - 17.2.7-847-g45317f95 b333581ccdf5 0824e2621be5 2024-04-16T05:44:51.464 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (4m) 20s ago 4m 12.2M - 17.2.7-847-g45317f95 b333581ccdf5 b332a72cc594 2024-04-16T05:44:51.464 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (4m) 23s ago 4m 12.4M - 17.2.7-847-g45317f95 b333581ccdf5 f5e36815ddfa 2024-04-16T05:44:51.464 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (4m) 23s ago 4m 12.4M - 17.2.7-847-g45317f95 b333581ccdf5 755a73c65a7f 2024-04-16T05:44:51.464 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:8443,9283,8765 running (43s) 20s ago 13m 477M - 19.0.0-2884-g218a6914 a4e4811815e4 b69a0b08e12b 2024-04-16T05:44:51.464 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283 running (2m) 23s ago 10m 75.1M - 19.0.0-2884-g218a6914 a4e4811815e4 b1a1b7a6d536 2024-04-16T05:44:51.464 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (13m) 20s ago 13m 52.7M 2048M 17.2.7-847-g45317f95 b333581ccdf5 88575dbb8d98 2024-04-16T05:44:51.464 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (10m) 23s ago 10m 40.1M 2048M 17.2.7-847-g45317f95 b333581ccdf5 5df19eab6d21 2024-04-16T05:44:51.464 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (78s) 20s ago 12m 7587k - 1.5.0 0da6a335fe13 5e29e29868b0 2024-04-16T05:44:51.464 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (52s) 23s ago 10m 7562k - 1.5.0 0da6a335fe13 a360922bc621 2024-04-16T05:44:51.464 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (9m) 20s ago 9m 46.9M 1557M 17.2.7-847-g45317f95 b333581ccdf5 15e049573bca 2024-04-16T05:44:51.464 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (8m) 20s ago 8m 46.8M 1557M 17.2.7-847-g45317f95 b333581ccdf5 e146295f6500 2024-04-16T05:44:51.464 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (8m) 20s ago 8m 48.2M 1557M 17.2.7-847-g45317f95 b333581ccdf5 6212f6a18fa4 2024-04-16T05:44:51.465 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (7m) 23s ago 7m 44.8M 2581M 17.2.7-847-g45317f95 b333581ccdf5 d6724ad05f9f 2024-04-16T05:44:51.465 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (7m) 23s ago 7m 43.7M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7a59908a3e10 2024-04-16T05:44:51.465 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (6m) 23s ago 6m 43.2M 2581M 17.2.7-847-g45317f95 b333581ccdf5 8e4b60163907 2024-04-16T05:44:51.465 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 starting - - - - 2024-04-16T05:44:51.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.468+0000 7ffb06200700 1 -- 172.21.15.27:0/3941456931 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7ffae806e620 msgr2=0x7ffae8070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:51.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.468+0000 7ffb06200700 1 --2- 172.21.15.27:0/3941456931 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7ffae806e620 0x7ffae8070ad0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ffaf000b3b0 tx=0x7ffaf0009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:51.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.468+0000 7ffb06200700 1 -- 172.21.15.27:0/3941456931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb000708f0 msgr2=0x7ffb00113f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:51.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.468+0000 7ffb06200700 1 --2- 172.21.15.27:0/3941456931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb000708f0 0x7ffb00113f10 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ffaf400b790 tx=0x7ffaf4009220 comp rx=0 tx=0).stop 2024-04-16T05:44:51.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.468+0000 7ffb06200700 1 -- 172.21.15.27:0/3941456931 shutdown_connections 2024-04-16T05:44:51.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.468+0000 7ffb06200700 1 --2- 172.21.15.27:0/3941456931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffb000708f0 0x7ffb00113f10 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:51.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.468+0000 7ffb06200700 1 --2- 172.21.15.27:0/3941456931 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7ffae806e620 0x7ffae8070ad0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:51.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.468+0000 7ffb06200700 1 --2- 172.21.15.27:0/3941456931 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb00071ea0 0x7ffb00114450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:51.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.468+0000 7ffb06200700 1 -- 172.21.15.27:0/3941456931 >> 172.21.15.27:0/3941456931 conn(0x7ffb000f83d0 msgr2=0x7ffb000f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:51.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.468+0000 7ffb06200700 1 -- 172.21.15.27:0/3941456931 shutdown_connections 2024-04-16T05:44:51.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.468+0000 7ffb06200700 1 -- 172.21.15.27:0/3941456931 wait complete. 2024-04-16T05:44:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.618+0000 7fbd81000700 1 -- 172.21.15.27:0/3284292934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd7c066290 msgr2=0x7fbd7c101e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.618+0000 7fbd81000700 1 --2- 172.21.15.27:0/3284292934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd7c066290 0x7fbd7c101e90 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c004940 tx=0x7fbd6c009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.619+0000 7fbd81000700 1 -- 172.21.15.27:0/3284292934 shutdown_connections 2024-04-16T05:44:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.619+0000 7fbd81000700 1 --2- 172.21.15.27:0/3284292934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd7c1023d0 0x7fbd7c1047b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.619+0000 7fbd81000700 1 --2- 172.21.15.27:0/3284292934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd7c066290 0x7fbd7c101e90 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.619+0000 7fbd81000700 1 -- 172.21.15.27:0/3284292934 >> 172.21.15.27:0/3284292934 conn(0x7fbd7c0f74c0 msgr2=0x7fbd7c0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.619+0000 7fbd81000700 1 -- 172.21.15.27:0/3284292934 shutdown_connections 2024-04-16T05:44:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.619+0000 7fbd81000700 1 -- 172.21.15.27:0/3284292934 wait complete. 2024-04-16T05:44:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.620+0000 7fbd81000700 1 Processor -- start 2024-04-16T05:44:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.620+0000 7fbd81000700 1 -- start start 2024-04-16T05:44:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.620+0000 7fbd81000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd7c066290 0x7fbd7c10fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.620+0000 7fbd81000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd7c1023d0 0x7fbd7c110150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.620+0000 7fbd81000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7c110750 con 0x7fbd7c1023d0 2024-04-16T05:44:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.620+0000 7fbd81000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7c110f50 con 0x7fbd7c066290 2024-04-16T05:44:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.620+0000 7fbd79400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd7c1023d0 0x7fbd7c110150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.621+0000 7fbd79400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd7c1023d0 0x7fbd7c110150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36076/0 (socket says 172.21.15.27:36076) 2024-04-16T05:44:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.621+0000 7fbd79400700 1 -- 172.21.15.27:0/95153006 learned_addr learned my addr 172.21.15.27:0/95153006 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:44:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.621+0000 7fbd79e00700 1 --2- 172.21.15.27:0/95153006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd7c066290 0x7fbd7c10fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.621+0000 7fbd79400700 1 -- 172.21.15.27:0/95153006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd7c066290 msgr2=0x7fbd7c10fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.621+0000 7fbd79400700 1 --2- 172.21.15.27:0/95153006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd7c066290 0x7fbd7c10fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.621+0000 7fbd79400700 1 -- 172.21.15.27:0/95153006 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd6c005040 con 0x7fbd7c1023d0 2024-04-16T05:44:51.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.621+0000 7fbd79400700 1 --2- 172.21.15.27:0/95153006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd7c1023d0 0x7fbd7c110150 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fbd68002a80 tx=0x7fbd680046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:51.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.622+0000 7fbd72000700 1 -- 172.21.15.27:0/95153006 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd68007cc0 con 0x7fbd7c1023d0 2024-04-16T05:44:51.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.622+0000 7fbd72000700 1 -- 172.21.15.27:0/95153006 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd6800f040 con 0x7fbd7c1023d0 2024-04-16T05:44:51.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.622+0000 7fbd81000700 1 -- 172.21.15.27:0/95153006 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd7c111230 con 0x7fbd7c1023d0 2024-04-16T05:44:51.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.622+0000 7fbd72000700 1 -- 172.21.15.27:0/95153006 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbd6800dac0 con 0x7fbd7c1023d0 2024-04-16T05:44:51.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.622+0000 7fbd81000700 1 -- 172.21.15.27:0/95153006 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd7c111770 con 0x7fbd7c1023d0 2024-04-16T05:44:51.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.623+0000 7fbd81000700 1 -- 172.21.15.27:0/95153006 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd7c05f5f0 con 0x7fbd7c1023d0 2024-04-16T05:44:51.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.624+0000 7fbd72000700 1 -- 172.21.15.27:0/95153006 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 31) v1 ==== 92569+0+0 (secure 0 0 0) 0x7fbd6800dc20 con 0x7fbd7c1023d0 2024-04-16T05:44:51.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.625+0000 7fbd72000700 1 --2- 172.21.15.27:0/95153006 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7fbd6006e620 0x7fbd60070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:51.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.625+0000 7fbd72000700 1 -- 172.21.15.27:0/95153006 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fbd6808ddd0 con 0x7fbd7c1023d0 2024-04-16T05:44:51.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.625+0000 7fbd79e00700 1 --2- 172.21.15.27:0/95153006 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7fbd6006e620 0x7fbd60070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:51.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.626+0000 7fbd79e00700 1 --2- 172.21.15.27:0/95153006 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7fbd6006e620 0x7fbd60070ad0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c00b3b0 tx=0x7fbd6c009e30 comp rx=0 tx=0).ready entity=mgr.14716 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:51.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.630+0000 7fbd72000700 1 -- 172.21.15.27:0/95153006 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fbd6805acc0 con 0x7fbd7c1023d0 2024-04-16T05:44:51.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.947+0000 7fbd81000700 1 -- 172.21.15.27:0/95153006 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fbd7c1119e0 con 0x7fbd7c1023d0 2024-04-16T05:44:51.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.949+0000 7fbd72000700 1 -- 172.21.15.27:0/95153006 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7fbd6805a850 con 0x7fbd7c1023d0 2024-04-16T05:44:51.947 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:44:51.947 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:44:51.947 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2 2024-04-16T05:44:51.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:44:51.947 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:44:51.947 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:44:51.947 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:44:51.947 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:44:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T05:44:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:44:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:44:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T05:44:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:44:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:44:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 12, 2024-04-16T05:44:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:44:51.948 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:44:51.948 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:44:51.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.951+0000 7fbd81000700 1 -- 172.21.15.27:0/95153006 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7fbd6006e620 msgr2=0x7fbd60070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:51.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.951+0000 7fbd81000700 1 --2- 172.21.15.27:0/95153006 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7fbd6006e620 0x7fbd60070ad0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c00b3b0 tx=0x7fbd6c009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:51.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.951+0000 7fbd81000700 1 -- 172.21.15.27:0/95153006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd7c1023d0 msgr2=0x7fbd7c110150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:51.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.951+0000 7fbd81000700 1 --2- 172.21.15.27:0/95153006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd7c1023d0 0x7fbd7c110150 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fbd68002a80 tx=0x7fbd680046b0 comp rx=0 tx=0).stop 2024-04-16T05:44:51.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.951+0000 7fbd81000700 1 -- 172.21.15.27:0/95153006 shutdown_connections 2024-04-16T05:44:51.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.951+0000 7fbd81000700 1 --2- 172.21.15.27:0/95153006 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7fbd6006e620 0x7fbd60070ad0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:51.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.951+0000 7fbd81000700 1 --2- 172.21.15.27:0/95153006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd7c066290 0x7fbd7c10fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:51.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.951+0000 7fbd81000700 1 --2- 172.21.15.27:0/95153006 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd7c1023d0 0x7fbd7c110150 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:51.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.951+0000 7fbd81000700 1 -- 172.21.15.27:0/95153006 >> 172.21.15.27:0/95153006 conn(0x7fbd7c0f74c0 msgr2=0x7fbd7c0f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:51.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.952+0000 7fbd81000700 1 -- 172.21.15.27:0/95153006 shutdown_connections 2024-04-16T05:44:51.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:51.952+0000 7fbd81000700 1 -- 172.21.15.27:0/95153006 wait complete. 2024-04-16T05:44:52.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.108+0000 7f1715a00700 1 -- 172.21.15.27:0/2859045128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1710104520 msgr2=0x7f1710104990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:52.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.108+0000 7f1715a00700 1 --2- 172.21.15.27:0/2859045128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1710104520 0x7f1710104990 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f16f8004940 tx=0x7f16f8009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:52.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.109+0000 7f1715a00700 1 -- 172.21.15.27:0/2859045128 shutdown_connections 2024-04-16T05:44:52.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.109+0000 7f1715a00700 1 --2- 172.21.15.27:0/2859045128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1710104520 0x7f1710104990 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:52.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.109+0000 7f1715a00700 1 --2- 172.21.15.27:0/2859045128 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1710103270 0x7f1710103680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:52.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.109+0000 7f1715a00700 1 -- 172.21.15.27:0/2859045128 >> 172.21.15.27:0/2859045128 conn(0x7f1710072530 msgr2=0x7f1710074940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:52.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.109+0000 7f1715a00700 1 -- 172.21.15.27:0/2859045128 shutdown_connections 2024-04-16T05:44:52.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.109+0000 7f1715a00700 1 -- 172.21.15.27:0/2859045128 wait complete. 2024-04-16T05:44:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.110+0000 7f1715a00700 1 Processor -- start 2024-04-16T05:44:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.110+0000 7f1715a00700 1 -- start start 2024-04-16T05:44:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.110+0000 7f1715a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1710103270 0x7f1710113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.110+0000 7f1715a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1710104520 0x7f1710114460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.110+0000 7f1715a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1710114a60 con 0x7f1710104520 2024-04-16T05:44:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.110+0000 7f1715a00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17101151a0 con 0x7f1710103270 2024-04-16T05:44:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.110+0000 7f1707e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1710104520 0x7f1710114460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.111+0000 7f1707e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1710104520 0x7f1710114460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36096/0 (socket says 172.21.15.27:36096) 2024-04-16T05:44:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.111+0000 7f1707e00700 1 -- 172.21.15.27:0/3819300058 learned_addr learned my addr 172.21.15.27:0/3819300058 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:44:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.111+0000 7f170fe00700 1 --2- 172.21.15.27:0/3819300058 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1710103270 0x7f1710113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.111+0000 7f1707e00700 1 -- 172.21.15.27:0/3819300058 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1710103270 msgr2=0x7f1710113f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.111+0000 7f1707e00700 1 --2- 172.21.15.27:0/3819300058 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1710103270 0x7f1710113f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:52.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.111+0000 7f1707e00700 1 -- 172.21.15.27:0/3819300058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f8005040 con 0x7f1710104520 2024-04-16T05:44:52.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.111+0000 7f1707e00700 1 --2- 172.21.15.27:0/3819300058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1710104520 0x7f1710114460 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009150 tx=0x7f16f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:52.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.112+0000 7f170cc00700 1 -- 172.21.15.27:0/3819300058 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16f800e040 con 0x7f1710104520 2024-04-16T05:44:52.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.112+0000 7f170cc00700 1 -- 172.21.15.27:0/3819300058 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f16f800bbf0 con 0x7f1710104520 2024-04-16T05:44:52.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.112+0000 7f1715a00700 1 -- 172.21.15.27:0/3819300058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1710115420 con 0x7f1710104520 2024-04-16T05:44:52.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.112+0000 7f170cc00700 1 -- 172.21.15.27:0/3819300058 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16f801c630 con 0x7f1710104520 2024-04-16T05:44:52.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.112+0000 7f1715a00700 1 -- 172.21.15.27:0/3819300058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17101159e0 con 0x7f1710104520 2024-04-16T05:44:52.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.113+0000 7f1715a00700 1 -- 172.21.15.27:0/3819300058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17100760a0 con 0x7f1710104520 2024-04-16T05:44:52.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.114+0000 7f170cc00700 1 -- 172.21.15.27:0/3819300058 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 31) v1 ==== 92569+0+0 (secure 0 0 0) 0x7f16f800bd60 con 0x7f1710104520 2024-04-16T05:44:52.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.115+0000 7f170cc00700 1 --2- 172.21.15.27:0/3819300058 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f16f406e630 0x7f16f4070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:52.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.115+0000 7f170cc00700 1 -- 172.21.15.27:0/3819300058 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f16f808f580 con 0x7f1710104520 2024-04-16T05:44:52.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.115+0000 7f170fe00700 1 --2- 172.21.15.27:0/3819300058 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f16f406e630 0x7f16f4070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:52.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.116+0000 7f170fe00700 1 --2- 172.21.15.27:0/3819300058 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f16f406e630 0x7f16f4070ae0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1700004f40 tx=0x7f170000b040 comp rx=0 tx=0).ready entity=mgr.14716 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:52.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.120+0000 7f170cc00700 1 -- 172.21.15.27:0/3819300058 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f16f805c470 con 0x7f1710104520 2024-04-16T05:44:52.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:52 smithi122 ceph-mon[35998]: from='client.14774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:44:52.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:52 smithi122 ceph-mon[35998]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T05:44:52.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:52 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/95153006' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:44:52.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:52 smithi027 ceph-mon[29206]: from='client.14774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:44:52.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:52 smithi027 ceph-mon[29206]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T05:44:52.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:52 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/95153006' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:44:52.385 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.387+0000 7f1715a00700 1 -- 172.21.15.27:0/3819300058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f171004bed0 con 0x7f1710104520 2024-04-16T05:44:52.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.388+0000 7f170cc00700 1 -- 172.21.15.27:0/3819300058 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1810 (secure 0 0 0) 0x7f16f8013e60 con 0x7f1710104520 2024-04-16T05:44:52.387 INFO:teuthology.orchestra.run.smithi027.stdout:e16 2024-04-16T05:44:52.387 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:44:52.387 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:44:52.387 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:44:52.387 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:44:52.387 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:44:52.387 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:44:52.387 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 15 2024-04-16T05:44:52.387 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:44:52.388 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:44:52.388 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:40:32.701649+0000 2024-04-16T05:44:52.388 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:44:52.388 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:44:52.388 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:44:52.388 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:44:52.388 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:44:52.388 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:44:52.388 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:44:52.388 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 46 2024-04-16T05:44:52.388 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:44:52.388 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:44:52.388 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:44:52.388 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=14462} 2024-04-16T05:44:52.389 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:44:52.389 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:44:52.389 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:44:52.389 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:44:52.389 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:44:52.389 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:44:52.389 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:44:52.389 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:44:52.389 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:44:52.389 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:44:52.389 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:44:52.389 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:44:52.389 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:44:52.389 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{-1:14474} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2065690722,v1:172.21.15.27:6827/2065690722] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:44:52.390 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:24263} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:44:52.390 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:24273} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:44:52.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.392+0000 7f1715a00700 1 -- 172.21.15.27:0/3819300058 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f16f406e630 msgr2=0x7f16f4070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:52.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.392+0000 7f1715a00700 1 --2- 172.21.15.27:0/3819300058 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f16f406e630 0x7f16f4070ae0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1700004f40 tx=0x7f170000b040 comp rx=0 tx=0).stop 2024-04-16T05:44:52.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.392+0000 7f1715a00700 1 -- 172.21.15.27:0/3819300058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1710104520 msgr2=0x7f1710114460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:52.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.392+0000 7f1715a00700 1 --2- 172.21.15.27:0/3819300058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1710104520 0x7f1710114460 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009150 tx=0x7f16f8009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:52.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.392+0000 7f1715a00700 1 -- 172.21.15.27:0/3819300058 shutdown_connections 2024-04-16T05:44:52.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.392+0000 7f1715a00700 1 --2- 172.21.15.27:0/3819300058 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f16f406e630 0x7f16f4070ae0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:52.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.392+0000 7f1715a00700 1 --2- 172.21.15.27:0/3819300058 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1710103270 0x7f1710113f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:52.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.392+0000 7f1715a00700 1 --2- 172.21.15.27:0/3819300058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1710104520 0x7f1710114460 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:52.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.392+0000 7f1715a00700 1 -- 172.21.15.27:0/3819300058 >> 172.21.15.27:0/3819300058 conn(0x7f1710072530 msgr2=0x7f1710074b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:52.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.392+0000 7f1715a00700 1 -- 172.21.15.27:0/3819300058 shutdown_connections 2024-04-16T05:44:52.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.392+0000 7f1715a00700 1 -- 172.21.15.27:0/3819300058 wait complete. 2024-04-16T05:44:52.392 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2024-04-16T05:44:52.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.545+0000 7f04e2200700 1 -- 172.21.15.27:0/3028330355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04dc0ff020 msgr2=0x7f04dc0ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:52.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.545+0000 7f04e2200700 1 --2- 172.21.15.27:0/3028330355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04dc0ff020 0x7f04dc0ff430 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f04cc004940 tx=0x7f04cc009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:52.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.546+0000 7f04e2200700 1 -- 172.21.15.27:0/3028330355 shutdown_connections 2024-04-16T05:44:52.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.546+0000 7f04e2200700 1 --2- 172.21.15.27:0/3028330355 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04dc1002d0 0x7f04dc100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:52.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.546+0000 7f04e2200700 1 --2- 172.21.15.27:0/3028330355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04dc0ff020 0x7f04dc0ff430 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:52.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.546+0000 7f04e2200700 1 -- 172.21.15.27:0/3028330355 >> 172.21.15.27:0/3028330355 conn(0x7f04dc0fa950 msgr2=0x7f04dc0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:52.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.546+0000 7f04e2200700 1 -- 172.21.15.27:0/3028330355 shutdown_connections 2024-04-16T05:44:52.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.546+0000 7f04e2200700 1 -- 172.21.15.27:0/3028330355 wait complete. 2024-04-16T05:44:52.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.547+0000 7f04e2200700 1 Processor -- start 2024-04-16T05:44:52.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.547+0000 7f04e2200700 1 -- start start 2024-04-16T05:44:52.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.548+0000 7f04e2200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04dc0ff020 0x7f04dc10fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:52.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.548+0000 7f04e2200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04dc1002d0 0x7f04dc1100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:52.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.548+0000 7f04e2200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc1106a0 con 0x7f04dc1002d0 2024-04-16T05:44:52.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.548+0000 7f04e2200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc110810 con 0x7f04dc0ff020 2024-04-16T05:44:52.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.548+0000 7f04daa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04dc1002d0 0x7f04dc1100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:52.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.548+0000 7f04daa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04dc1002d0 0x7f04dc1100a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36108/0 (socket says 172.21.15.27:36108) 2024-04-16T05:44:52.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.548+0000 7f04daa00700 1 -- 172.21.15.27:0/625166451 learned_addr learned my addr 172.21.15.27:0/625166451 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:44:52.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.548+0000 7f04db400700 1 --2- 172.21.15.27:0/625166451 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04dc0ff020 0x7f04dc10fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:52.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.548+0000 7f04daa00700 1 -- 172.21.15.27:0/625166451 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04dc0ff020 msgr2=0x7f04dc10fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:52.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.548+0000 7f04daa00700 1 --2- 172.21.15.27:0/625166451 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04dc0ff020 0x7f04dc10fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:52.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.548+0000 7f04daa00700 1 -- 172.21.15.27:0/625166451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04cc005040 con 0x7f04dc1002d0 2024-04-16T05:44:52.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.548+0000 7f04db400700 1 --2- 172.21.15.27:0/625166451 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04dc0ff020 0x7f04dc10fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:44:52.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.549+0000 7f04daa00700 1 --2- 172.21.15.27:0/625166451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04dc1002d0 0x7f04dc1100a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f04c800c8c0 tx=0x7f04c80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:52.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.549+0000 7f04d3400700 1 -- 172.21.15.27:0/625166451 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04c800a980 con 0x7f04dc1002d0 2024-04-16T05:44:52.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.549+0000 7f04d3400700 1 -- 172.21.15.27:0/625166451 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f04c800ce60 con 0x7f04dc1002d0 2024-04-16T05:44:52.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.549+0000 7f04e2200700 1 -- 172.21.15.27:0/625166451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04dc115270 con 0x7f04dc1002d0 2024-04-16T05:44:52.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.549+0000 7f04d3400700 1 -- 172.21.15.27:0/625166451 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04c80115e0 con 0x7f04dc1002d0 2024-04-16T05:44:52.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.549+0000 7f04e2200700 1 -- 172.21.15.27:0/625166451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04dc115830 con 0x7f04dc1002d0 2024-04-16T05:44:52.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.550+0000 7f04e2200700 1 -- 172.21.15.27:0/625166451 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04dc04c710 con 0x7f04dc1002d0 2024-04-16T05:44:52.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.552+0000 7f04d3400700 1 -- 172.21.15.27:0/625166451 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 31) v1 ==== 92569+0+0 (secure 0 0 0) 0x7f04c8012400 con 0x7f04dc1002d0 2024-04-16T05:44:52.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.553+0000 7f04d3400700 1 --2- 172.21.15.27:0/625166451 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f04c006e630 0x7f04c0070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:52.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.553+0000 7f04d3400700 1 -- 172.21.15.27:0/625166451 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f04c808e080 con 0x7f04dc1002d0 2024-04-16T05:44:52.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.553+0000 7f04db400700 1 --2- 172.21.15.27:0/625166451 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f04c006e630 0x7f04c0070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:52.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.554+0000 7f04db400700 1 --2- 172.21.15.27:0/625166451 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f04c006e630 0x7f04c0070ae0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f04cc00b7a0 tx=0x7f04cc009e30 comp rx=0 tx=0).ready entity=mgr.14716 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:52.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.559+0000 7f04d3400700 1 -- 172.21.15.27:0/625166451 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7f04c805b1c0 con 0x7f04dc1002d0 2024-04-16T05:44:52.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.807+0000 7f04e2200700 1 -- 172.21.15.27:0/625166451 --> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f04dc104690 con 0x7f04c006e630 2024-04-16T05:44:52.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.809+0000 7f04d3400700 1 -- 172.21.15.27:0/625166451 <== mgr.14716 v2:172.21.15.27:6800/1090905394 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+368 (secure 0 0 0) 0x7f04dc104690 con 0x7f04c006e630 2024-04-16T05:44:52.808 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:44:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T05:44:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:44:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:44:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [ 2024-04-16T05:44:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr" 2024-04-16T05:44:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-16T05:44:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "2/21 daemons upgraded", 2024-04-16T05:44:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Currently upgrading mgr daemons", 2024-04-16T05:44:52.808 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:44:52.808 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:44:52.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.812+0000 7f04e2200700 1 -- 172.21.15.27:0/625166451 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f04c006e630 msgr2=0x7f04c0070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:52.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.812+0000 7f04e2200700 1 --2- 172.21.15.27:0/625166451 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f04c006e630 0x7f04c0070ae0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f04cc00b7a0 tx=0x7f04cc009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:52.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.812+0000 7f04e2200700 1 -- 172.21.15.27:0/625166451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04dc1002d0 msgr2=0x7f04dc1100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:52.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.812+0000 7f04e2200700 1 --2- 172.21.15.27:0/625166451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04dc1002d0 0x7f04dc1100a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f04c800c8c0 tx=0x7f04c80044c0 comp rx=0 tx=0).stop 2024-04-16T05:44:52.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.812+0000 7f04e2200700 1 -- 172.21.15.27:0/625166451 shutdown_connections 2024-04-16T05:44:52.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.812+0000 7f04e2200700 1 --2- 172.21.15.27:0/625166451 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f04c006e630 0x7f04c0070ae0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:52.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.812+0000 7f04e2200700 1 --2- 172.21.15.27:0/625166451 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04dc0ff020 0x7f04dc10fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:52.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.812+0000 7f04e2200700 1 --2- 172.21.15.27:0/625166451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f04dc1002d0 0x7f04dc1100a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:52.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.812+0000 7f04e2200700 1 -- 172.21.15.27:0/625166451 >> 172.21.15.27:0/625166451 conn(0x7f04dc0fa950 msgr2=0x7f04dc0fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:52.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.813+0000 7f04e2200700 1 -- 172.21.15.27:0/625166451 shutdown_connections 2024-04-16T05:44:52.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.813+0000 7f04e2200700 1 -- 172.21.15.27:0/625166451 wait complete. 2024-04-16T05:44:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.966+0000 7fa4a5400700 1 -- 172.21.15.27:0/244310976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4a01046d0 msgr2=0x7fa4a0104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.966+0000 7fa4a5400700 1 --2- 172.21.15.27:0/244310976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4a01046d0 0x7fa4a0104b40 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa488004940 tx=0x7fa488009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.966+0000 7fa4a5400700 1 -- 172.21.15.27:0/244310976 shutdown_connections 2024-04-16T05:44:52.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.966+0000 7fa4a5400700 1 --2- 172.21.15.27:0/244310976 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4a01046d0 0x7fa4a0104b40 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:52.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.966+0000 7fa4a5400700 1 --2- 172.21.15.27:0/244310976 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4a0103420 0x7fa4a0103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:52.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.966+0000 7fa4a5400700 1 -- 172.21.15.27:0/244310976 >> 172.21.15.27:0/244310976 conn(0x7fa4a0072640 msgr2=0x7fa4a0074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:52.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.966+0000 7fa4a5400700 1 -- 172.21.15.27:0/244310976 shutdown_connections 2024-04-16T05:44:52.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.966+0000 7fa4a5400700 1 -- 172.21.15.27:0/244310976 wait complete. 2024-04-16T05:44:52.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.967+0000 7fa4a5400700 1 Processor -- start 2024-04-16T05:44:52.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.967+0000 7fa4a5400700 1 -- start start 2024-04-16T05:44:52.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.967+0000 7fa4a5400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4a0103420 0x7fa4a0113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:52.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.967+0000 7fa4a5400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4a01046d0 0x7fa4a0114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:52.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.967+0000 7fa4a5400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a0114a90 con 0x7fa4a01046d0 2024-04-16T05:44:52.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.967+0000 7fa4a5400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a0114c00 con 0x7fa4a0103420 2024-04-16T05:44:52.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.968+0000 7fa495e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4a01046d0 0x7fa4a0114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:52.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.968+0000 7fa495e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4a01046d0 0x7fa4a0114490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36120/0 (socket says 172.21.15.27:36120) 2024-04-16T05:44:52.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.968+0000 7fa495e00700 1 -- 172.21.15.27:0/2057797046 learned_addr learned my addr 172.21.15.27:0/2057797046 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:44:52.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.968+0000 7fa49e800700 1 --2- 172.21.15.27:0/2057797046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4a0103420 0x7fa4a0113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:52.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.968+0000 7fa495e00700 1 -- 172.21.15.27:0/2057797046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4a0103420 msgr2=0x7fa4a0113f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:52.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.968+0000 7fa495e00700 1 --2- 172.21.15.27:0/2057797046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4a0103420 0x7fa4a0113f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:52.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.968+0000 7fa495e00700 1 -- 172.21.15.27:0/2057797046 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa488005040 con 0x7fa4a01046d0 2024-04-16T05:44:52.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.969+0000 7fa495e00700 1 --2- 172.21.15.27:0/2057797046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4a01046d0 0x7fa4a0114490 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa48800b3b0 tx=0x7fa488009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:52.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.969+0000 7fa497400700 1 -- 172.21.15.27:0/2057797046 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa48800e040 con 0x7fa4a01046d0 2024-04-16T05:44:52.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.969+0000 7fa497400700 1 -- 172.21.15.27:0/2057797046 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa488004290 con 0x7fa4a01046d0 2024-04-16T05:44:52.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.969+0000 7fa4a5400700 1 -- 172.21.15.27:0/2057797046 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a0119600 con 0x7fa4a01046d0 2024-04-16T05:44:52.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.969+0000 7fa497400700 1 -- 172.21.15.27:0/2057797046 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa48801c630 con 0x7fa4a01046d0 2024-04-16T05:44:52.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.969+0000 7fa4a5400700 1 -- 172.21.15.27:0/2057797046 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a0119bc0 con 0x7fa4a01046d0 2024-04-16T05:44:52.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.970+0000 7fa4a5400700 1 -- 172.21.15.27:0/2057797046 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a004c710 con 0x7fa4a01046d0 2024-04-16T05:44:52.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.971+0000 7fa497400700 1 -- 172.21.15.27:0/2057797046 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 31) v1 ==== 92569+0+0 (secure 0 0 0) 0x7fa48800bbf0 con 0x7fa4a01046d0 2024-04-16T05:44:52.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.972+0000 7fa497400700 1 --2- 172.21.15.27:0/2057797046 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7fa48406e630 0x7fa484070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:44:52.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.972+0000 7fa497400700 1 -- 172.21.15.27:0/2057797046 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7fa48808f660 con 0x7fa4a01046d0 2024-04-16T05:44:52.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.972+0000 7fa49e800700 1 --2- 172.21.15.27:0/2057797046 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7fa48406e630 0x7fa484070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:44:52.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.973+0000 7fa49e800700 1 --2- 172.21.15.27:0/2057797046 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7fa48406e630 0x7fa484070ae0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa490007fd0 tx=0x7fa490009040 comp rx=0 tx=0).ready entity=mgr.14716 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:44:52.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:52.977+0000 7fa497400700 1 -- 172.21.15.27:0/2057797046 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186652 (secure 0 0 0) 0x7fa48805c550 con 0x7fa4a01046d0 2024-04-16T05:44:53.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:53 smithi027 ceph-mon[29206]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:44:53.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:53 smithi027 ceph-mon[29206]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:44:53.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:53 smithi027 ceph-mon[29206]: pgmap v17: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:53.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:53 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/3819300058' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:44:53.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:53 smithi122 ceph-mon[35998]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:44:53.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:53 smithi122 ceph-mon[35998]: from='client.14782 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:44:53.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:53 smithi122 ceph-mon[35998]: pgmap v17: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:53.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:53 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/3819300058' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:44:53.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:53.299+0000 7fa4a5400700 1 -- 172.21.15.27:0/2057797046 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa4a0119ea0 con 0x7fa4a01046d0 2024-04-16T05:44:53.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:53.300+0000 7fa497400700 1 -- 172.21.15.27:0/2057797046 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fa48805c0e0 con 0x7fa4a01046d0 2024-04-16T05:44:53.298 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:44:53.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:53.302+0000 7fa4a5400700 1 -- 172.21.15.27:0/2057797046 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7fa48406e630 msgr2=0x7fa484070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:53.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:53.302+0000 7fa4a5400700 1 --2- 172.21.15.27:0/2057797046 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7fa48406e630 0x7fa484070ae0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa490007fd0 tx=0x7fa490009040 comp rx=0 tx=0).stop 2024-04-16T05:44:53.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:53.302+0000 7fa4a5400700 1 -- 172.21.15.27:0/2057797046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4a01046d0 msgr2=0x7fa4a0114490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:44:53.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:53.302+0000 7fa4a5400700 1 --2- 172.21.15.27:0/2057797046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4a01046d0 0x7fa4a0114490 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa48800b3b0 tx=0x7fa488009e30 comp rx=0 tx=0).stop 2024-04-16T05:44:53.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:53.303+0000 7fa4a5400700 1 -- 172.21.15.27:0/2057797046 shutdown_connections 2024-04-16T05:44:53.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:53.303+0000 7fa4a5400700 1 --2- 172.21.15.27:0/2057797046 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7fa48406e630 0x7fa484070ae0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:53.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:53.303+0000 7fa4a5400700 1 --2- 172.21.15.27:0/2057797046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4a0103420 0x7fa4a0113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:53.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:53.303+0000 7fa4a5400700 1 --2- 172.21.15.27:0/2057797046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4a01046d0 0x7fa4a0114490 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:44:53.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:53.303+0000 7fa4a5400700 1 -- 172.21.15.27:0/2057797046 >> 172.21.15.27:0/2057797046 conn(0x7fa4a0072640 msgr2=0x7fa4a0073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:44:53.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:53.303+0000 7fa4a5400700 1 -- 172.21.15.27:0/2057797046 shutdown_connections 2024-04-16T05:44:53.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:44:53.303+0000 7fa4a5400700 1 -- 172.21.15.27:0/2057797046 wait complete. 2024-04-16T05:44:54.260 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:54 smithi122 ceph-mon[35998]: from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:44:54.260 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:54 smithi122 ceph-mon[35998]: Upgrade: Updating mgr.smithi122.sgokkq 2024-04-16T05:44:54.260 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:54 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.sgokkq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T05:44:54.260 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:54 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T05:44:54.260 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:54 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:44:54.260 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:54 smithi122 ceph-mon[35998]: Deploying daemon mgr.smithi122.sgokkq on smithi122 2024-04-16T05:44:54.260 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:54 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/2057797046' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:44:54.260 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:54 smithi122 ceph-mon[35998]: pgmap v18: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:54.260 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:54 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:44:54.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:54 smithi027 ceph-mon[29206]: from='client.14794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:44:54.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:54 smithi027 ceph-mon[29206]: Upgrade: Updating mgr.smithi122.sgokkq 2024-04-16T05:44:54.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:54 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.sgokkq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T05:44:54.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:54 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T05:44:54.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:54 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:44:54.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:54 smithi027 ceph-mon[29206]: Deploying daemon mgr.smithi122.sgokkq on smithi122 2024-04-16T05:44:54.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:54 smithi027 ceph-mon[29206]: from='client.? 172.21.15.27:0/2057797046' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:44:54.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:54 smithi027 ceph-mon[29206]: pgmap v18: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:54.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:54 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:44:56.906 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:56 smithi122 ceph-mon[35998]: pgmap v19: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:57.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:56 smithi027 ceph-mon[29206]: pgmap v19: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:59.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:58 smithi027 ceph-mon[29206]: pgmap v20: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:44:59.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:58 smithi122 ceph-mon[35998]: pgmap v20: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:00.032 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:59 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:00.032 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:59 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:00.032 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:44:59 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:45:00.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:59 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:00.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:59 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:00.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:44:59 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:45:00.893 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:00 smithi122 ceph-mon[35998]: pgmap v21: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:01.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:00 smithi027 ceph-mon[29206]: pgmap v21: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:03.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:02 smithi027 ceph-mon[29206]: pgmap v22: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:03.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:02 smithi122 ceph-mon[35998]: pgmap v22: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:04.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:03 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:04.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:03 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:04.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:03 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:04.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:03 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:05.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:05 smithi122 ceph-mon[35998]: pgmap v23: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:05.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:05 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:05.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:05 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:05.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:05 smithi027 ceph-mon[29206]: pgmap v23: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:05.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:05 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:05.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:05 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:06.332 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:06 smithi122 ceph-mon[35998]: pgmap v24: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:06.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:06 smithi027 ceph-mon[29206]: pgmap v24: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:07.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:07 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:07.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:07 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:07.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:07 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:07.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:07 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:08.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:08 smithi122 ceph-mon[35998]: pgmap v25: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:08.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:08 smithi122 ceph-mon[35998]: Standby manager daemon smithi122.sgokkq restarted 2024-04-16T05:45:08.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:08 smithi122 ceph-mon[35998]: Standby manager daemon smithi122.sgokkq started 2024-04-16T05:45:08.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:08 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.122:0/3062015308' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.sgokkq/crt"}]: dispatch 2024-04-16T05:45:08.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:08 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.122:0/3062015308' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T05:45:08.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:08 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.122:0/3062015308' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.sgokkq/key"}]: dispatch 2024-04-16T05:45:08.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:08 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.122:0/3062015308' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T05:45:08.438 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:08 smithi027 ceph-mon[29206]: pgmap v25: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:08.438 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:08 smithi027 ceph-mon[29206]: Standby manager daemon smithi122.sgokkq restarted 2024-04-16T05:45:08.438 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:08 smithi027 ceph-mon[29206]: Standby manager daemon smithi122.sgokkq started 2024-04-16T05:45:08.439 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:08 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.122:0/3062015308' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.sgokkq/crt"}]: dispatch 2024-04-16T05:45:08.439 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:08 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.122:0/3062015308' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T05:45:08.439 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:08 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.122:0/3062015308' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.sgokkq/key"}]: dispatch 2024-04-16T05:45:08.439 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:08 smithi027 ceph-mon[29206]: from='mgr.? 172.21.15.122:0/3062015308' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T05:45:08.621 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 124, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 73602 2024-04-16T05:45:08.621 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 124, success, outsize: 120 2024-04-16T05:45:08.653 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 126, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 73696 2024-04-16T05:45:08.653 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 126, success, outsize: 120 2024-04-16T05:45:08.678 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 128, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 73793 2024-04-16T05:45:08.678 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 128, success, outsize: 120 2024-04-16T05:45:08.704 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 130, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 73896 2024-04-16T05:45:08.704 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 130, success, outsize: 120 2024-04-16T05:45:08.728 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 132, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 73996 2024-04-16T05:45:08.729 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 132, success, outsize: 120 2024-04-16T05:45:08.751 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 134, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74090 2024-04-16T05:45:08.752 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 134, success, outsize: 120 2024-04-16T05:45:08.778 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 136, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74187 2024-04-16T05:45:08.778 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 136, success, outsize: 120 2024-04-16T05:45:08.803 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 138, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74290 2024-04-16T05:45:08.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 138, success, outsize: 120 2024-04-16T05:45:08.829 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 140, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74390 2024-04-16T05:45:08.830 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 140, success, outsize: 120 2024-04-16T05:45:08.854 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 142, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74484 2024-04-16T05:45:08.854 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 142, success, outsize: 120 2024-04-16T05:45:08.879 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 144, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74581 2024-04-16T05:45:08.879 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 144, success, outsize: 120 2024-04-16T05:45:08.906 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 146, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74684 2024-04-16T05:45:08.907 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 146, success, outsize: 120 2024-04-16T05:45:08.933 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 148, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74784 2024-04-16T05:45:08.933 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 148, success, outsize: 120 2024-04-16T05:45:08.956 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 150, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74878 2024-04-16T05:45:08.957 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 150, success, outsize: 120 2024-04-16T05:45:08.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 152, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 74975 2024-04-16T05:45:08.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 152, success, outsize: 120 2024-04-16T05:45:09.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 154, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75078 2024-04-16T05:45:09.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 154, success, outsize: 120 2024-04-16T05:45:09.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 156, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75178 2024-04-16T05:45:09.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 156, success, outsize: 120 2024-04-16T05:45:09.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 158, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75272 2024-04-16T05:45:09.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 158, success, outsize: 120 2024-04-16T05:45:09.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 160, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75369 2024-04-16T05:45:09.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 160, success, outsize: 120 2024-04-16T05:45:09.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 162, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75472 2024-04-16T05:45:09.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 162, success, outsize: 120 2024-04-16T05:45:09.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 164, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75572 2024-04-16T05:45:09.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 164, success, outsize: 120 2024-04-16T05:45:09.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 166, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75666 2024-04-16T05:45:09.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 166, success, outsize: 120 2024-04-16T05:45:09.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 168, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75763 2024-04-16T05:45:09.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 168, success, outsize: 120 2024-04-16T05:45:09.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 170, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75866 2024-04-16T05:45:09.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 170, success, outsize: 120 2024-04-16T05:45:09.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 172, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 75966 2024-04-16T05:45:09.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 172, success, outsize: 120 2024-04-16T05:45:09.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 174, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 76060 2024-04-16T05:45:09.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 174, success, outsize: 120 2024-04-16T05:45:09.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 176, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 76157 2024-04-16T05:45:09.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 176, success, outsize: 120 2024-04-16T05:45:09.285 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 178, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 76260 2024-04-16T05:45:09.286 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 178, success, outsize: 120 2024-04-16T05:45:09.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:09 smithi122 ceph-mon[35998]: mgrmap e32: smithi027.qgzypi(active, since 44s), standbys: smithi122.sgokkq 2024-04-16T05:45:09.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:09 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:45:09.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:09 smithi027 ceph-mon[29206]: mgrmap e32: smithi027.qgzypi(active, since 44s), standbys: smithi122.sgokkq 2024-04-16T05:45:09.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:09 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:45:10.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:10 smithi122 ceph-mon[35998]: pgmap v26: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:10.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:10 smithi027 ceph-mon[29206]: pgmap v26: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:11.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:11 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:11.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:11 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:11.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:11 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:45:11.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:11 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:45:11.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:11 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:11.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:11 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:45:11.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:11 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:45:11.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:11 smithi122 ceph-mon[35998]: Upgrade: Setting container_image for all mgr 2024-04-16T05:45:11.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:11 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:11.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:11 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi027.qgzypi"}]: dispatch 2024-04-16T05:45:11.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:11 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi027.qgzypi"}]': finished 2024-04-16T05:45:11.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:11 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi122.sgokkq"}]: dispatch 2024-04-16T05:45:11.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:11 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi122.sgokkq"}]': finished 2024-04-16T05:45:11.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:11 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-16T05:45:11.548 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:11 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:11.548 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:11 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:11.548 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:11 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:45:11.548 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:11 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:45:11.548 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:11 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:11.548 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:11 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:45:11.548 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:11 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:45:11.549 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:11 smithi027 ceph-mon[29206]: Upgrade: Setting container_image for all mgr 2024-04-16T05:45:11.549 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:11 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:11.549 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:11 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi027.qgzypi"}]: dispatch 2024-04-16T05:45:11.549 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:11 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi027.qgzypi"}]': finished 2024-04-16T05:45:11.549 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:11 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi122.sgokkq"}]: dispatch 2024-04-16T05:45:11.549 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:11 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi122.sgokkq"}]': finished 2024-04-16T05:45:11.549 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:11 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-16T05:45:12.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:12 smithi122 ceph-mon[35998]: pgmap v27: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:12.551 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:12 smithi027 ceph-mon[29206]: pgmap v27: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:14.152 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:14 smithi027 ceph-mon[29206]: Upgrade: Updating mon.smithi027 2024-04-16T05:45:14.152 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:14 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:14.152 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:14 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T05:45:14.152 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:14 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T05:45:14.152 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:14 smithi027 ceph-mon[29206]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:45:14.152 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:14 smithi027 ceph-mon[29206]: Deploying daemon mon.smithi027 on smithi027 2024-04-16T05:45:14.152 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:14 smithi027 ceph-mon[29206]: pgmap v28: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:14.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:14 smithi122 ceph-mon[35998]: Upgrade: Updating mon.smithi027 2024-04-16T05:45:14.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:14 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:14.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:14 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T05:45:14.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:14 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T05:45:14.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:14 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:45:14.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:14 smithi122 ceph-mon[35998]: Deploying daemon mon.smithi027 on smithi027 2024-04-16T05:45:14.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:14 smithi122 ceph-mon[35998]: pgmap v28: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:16.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:15 smithi027 systemd[1]: Stopping Ceph mon.smithi027 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T05:45:16.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:16 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027[29201]: 2024-04-16T05:45:16.238+0000 7fad14400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi027 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T05:45:16.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:16 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027[29201]: 2024-04-16T05:45:16.238+0000 7fad14400700 -1 mon.smithi027@0(leader) e2 *** Got Signal Terminated *** 2024-04-16T05:45:16.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:16 smithi027 podman[93464]: 2024-04-16 05:45:16.485966373 +0000 UTC m=+0.388692005 container died 88575dbb8d98c7a63876cf20a632e21073d7d1bdc024885dd073ec093bc5600b (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-64502-quincy-3-45317f9, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T05:45:17.089 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:17 smithi027 podman[93464]: 2024-04-16 05:45:17.063504743 +0000 UTC m=+0.966230368 container cleanup 88575dbb8d98c7a63876cf20a632e21073d7d1bdc024885dd073ec093bc5600b (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-64502-quincy-3-45317f9, GIT_BRANCH=HEAD) 2024-04-16T05:45:17.089 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:17 smithi027 bash[93464]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027 2024-04-16T05:45:17.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:17 smithi027 podman[93489]: 2024-04-16 05:45:17.417636476 +0000 UTC m=+0.921609763 container remove 88575dbb8d98c7a63876cf20a632e21073d7d1bdc024885dd073ec093bc5600b (image=quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-16T05:45:18.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:17 smithi027 systemd[1]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@mon.smithi027.service: Deactivated successfully. 2024-04-16T05:45:18.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:17 smithi027 systemd[1]: Stopped Ceph mon.smithi027 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968. 2024-04-16T05:45:18.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:17 smithi027 systemd[1]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@mon.smithi027.service: Consumed 27.411s CPU time. 2024-04-16T05:45:18.658 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:18 smithi027 systemd[1]: Starting Ceph mon.smithi027 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T05:45:18.954 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:18 smithi027 podman[93617]: 2024-04-16 05:45:18.661069457 +0000 UTC m=+0.036158137 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T05:45:18.954 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:18 smithi027 podman[93617]: 2024-04-16 05:45:18.802562677 +0000 UTC m=+0.177651386 container create 299ca337e4ca2f01879e71aeb313a8b644c35532d69e0163715bc02d510b84ba (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-16T05:45:19.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 podman[93617]: 2024-04-16 05:45:19.04706759 +0000 UTC m=+0.422156322 container init 299ca337e4ca2f01879e71aeb313a8b644c35532d69e0163715bc02d510b84ba (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-16T05:45:19.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 podman[93617]: 2024-04-16 05:45:19.059287735 +0000 UTC m=+0.434376451 container start 299ca337e4ca2f01879e71aeb313a8b644c35532d69e0163715bc02d510b84ba (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-16T05:45:19.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: set uid:gid to 167:167 (ceph:ceph) 2024-04-16T05:45:19.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev), process ceph-mon, pid 2 2024-04-16T05:45:19.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: pidfile_write: ignore empty --pid-file 2024-04-16T05:45:19.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: load: jerasure load: lrc 2024-04-16T05:45:19.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: RocksDB version: 7.9.2 2024-04-16T05:45:19.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Git sha 0 2024-04-16T05:45:19.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Compile date 2024-04-12 04:47:35 2024-04-16T05:45:19.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: DB SUMMARY 2024-04-16T05:45:19.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: DB Session ID: 027YB5KU1BYLDJQGSCIX 2024-04-16T05:45:19.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: CURRENT file: CURRENT 2024-04-16T05:45:19.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: IDENTITY file: IDENTITY 2024-04-16T05:45:19.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1147 Bytes 2024-04-16T05:45:19.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi027/store.db dir, Total Num: 1, files: 000033.sst 2024-04-16T05:45:19.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi027/store.db: 000031.log size: 2144273 ; 2024-04-16T05:45:19.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.error_if_exists: 0 2024-04-16T05:45:19.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.create_if_missing: 0 2024-04-16T05:45:19.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.paranoid_checks: 1 2024-04-16T05:45:19.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-16T05:45:19.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-16T05:45:19.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-16T05:45:19.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.env: 0x5564af219cc0 2024-04-16T05:45:19.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.fs: PosixFileSystem 2024-04-16T05:45:19.310 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.info_log: 0x5564c4d93ae0 2024-04-16T05:45:19.310 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_file_opening_threads: 16 2024-04-16T05:45:19.310 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.statistics: (nil) 2024-04-16T05:45:19.310 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.use_fsync: 0 2024-04-16T05:45:19.310 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_log_file_size: 0 2024-04-16T05:45:19.310 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-16T05:45:19.310 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-16T05:45:19.310 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.keep_log_file_num: 1000 2024-04-16T05:45:19.310 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.recycle_log_file_num: 0 2024-04-16T05:45:19.310 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.allow_fallocate: 1 2024-04-16T05:45:19.310 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.allow_mmap_reads: 0 2024-04-16T05:45:19.311 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.allow_mmap_writes: 0 2024-04-16T05:45:19.311 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.use_direct_reads: 0 2024-04-16T05:45:19.311 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-16T05:45:19.311 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.create_missing_column_families: 0 2024-04-16T05:45:19.311 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.db_log_dir: 2024-04-16T05:45:19.311 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.wal_dir: 2024-04-16T05:45:19.311 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-16T05:45:19.311 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-16T05:45:19.311 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-16T05:45:19.311 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-16T05:45:19.311 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-16T05:45:19.311 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-16T05:45:19.311 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.advise_random_on_open: 1 2024-04-16T05:45:19.311 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.db_write_buffer_size: 0 2024-04-16T05:45:19.311 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.write_buffer_manager: 0x5564c4d641e0 2024-04-16T05:45:19.311 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-16T05:45:19.312 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-16T05:45:19.312 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-16T05:45:19.312 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.rate_limiter: (nil) 2024-04-16T05:45:19.312 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-16T05:45:19.312 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.wal_recovery_mode: 2 2024-04-16T05:45:19.312 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.enable_thread_tracking: 0 2024-04-16T05:45:19.312 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.enable_pipelined_write: 0 2024-04-16T05:45:19.312 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.unordered_write: 0 2024-04-16T05:45:19.312 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-16T05:45:19.312 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-16T05:45:19.312 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-16T05:45:19.312 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-16T05:45:19.312 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.row_cache: None 2024-04-16T05:45:19.312 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.wal_filter: None 2024-04-16T05:45:19.313 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-16T05:45:19.313 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.allow_ingest_behind: 0 2024-04-16T05:45:19.313 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.two_write_queues: 0 2024-04-16T05:45:19.313 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.manual_wal_flush: 0 2024-04-16T05:45:19.313 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.wal_compression: 0 2024-04-16T05:45:19.313 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.atomic_flush: 0 2024-04-16T05:45:19.313 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-16T05:45:19.313 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-16T05:45:19.313 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-16T05:45:19.313 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.log_readahead_size: 0 2024-04-16T05:45:19.313 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-16T05:45:19.313 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.best_efforts_recovery: 0 2024-04-16T05:45:19.313 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-16T05:45:19.313 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-16T05:45:19.313 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.allow_data_in_errors: 0 2024-04-16T05:45:19.314 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.db_host_id: __hostname__ 2024-04-16T05:45:19.314 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-16T05:45:19.314 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_background_jobs: 2 2024-04-16T05:45:19.314 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_background_compactions: -1 2024-04-16T05:45:19.314 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_subcompactions: 1 2024-04-16T05:45:19.314 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-16T05:45:19.314 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-16T05:45:19.314 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-16T05:45:19.314 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_total_wal_size: 0 2024-04-16T05:45:19.314 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-16T05:45:19.314 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-16T05:45:19.314 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-16T05:45:19.314 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-16T05:45:19.314 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_open_files: -1 2024-04-16T05:45:19.315 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.bytes_per_sync: 0 2024-04-16T05:45:19.315 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-16T05:45:19.315 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-16T05:45:19.315 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compaction_readahead_size: 0 2024-04-16T05:45:19.315 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_background_flushes: -1 2024-04-16T05:45:19.315 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Compression algorithms supported: 2024-04-16T05:45:19.315 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-16T05:45:19.315 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: kZSTD supported: 0 2024-04-16T05:45:19.315 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: kXpressCompression supported: 0 2024-04-16T05:45:19.315 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: kLZ4HCCompression supported: 1 2024-04-16T05:45:19.315 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: kZlibCompression supported: 1 2024-04-16T05:45:19.315 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: kSnappyCompression supported: 1 2024-04-16T05:45:19.316 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: kLZ4Compression supported: 1 2024-04-16T05:45:19.316 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: kBZip2Compression supported: 0 2024-04-16T05:45:19.316 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-16T05:45:19.316 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-16T05:45:19.316 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi027/store.db/MANIFEST-000015 2024-04-16T05:45:19.316 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-16T05:45:19.316 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-16T05:45:19.316 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.merge_operator: 2024-04-16T05:45:19.316 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compaction_filter: None 2024-04-16T05:45:19.317 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compaction_filter_factory: None 2024-04-16T05:45:19.317 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.sst_partitioner_factory: None 2024-04-16T05:45:19.317 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-16T05:45:19.317 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-16T05:45:19.317 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5564c4d93cc0) 2024-04-16T05:45:19.317 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: cache_index_and_filter_blocks: 1 2024-04-16T05:45:19.317 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-16T05:45:19.317 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-16T05:45:19.317 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: pin_top_level_index_and_filter: 1 2024-04-16T05:45:19.317 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: index_type: 0 2024-04-16T05:45:19.317 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: data_block_index_type: 0 2024-04-16T05:45:19.317 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: index_shortening: 1 2024-04-16T05:45:19.317 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-16T05:45:19.318 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: checksum: 4 2024-04-16T05:45:19.318 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: no_block_cache: 0 2024-04-16T05:45:19.318 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_cache: 0x5564c4d97090 2024-04-16T05:45:19.318 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_cache_name: BinnedLRUCache 2024-04-16T05:45:19.318 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_cache_options: 2024-04-16T05:45:19.318 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: capacity : 536870912 2024-04-16T05:45:19.318 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: num_shard_bits : 4 2024-04-16T05:45:19.318 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: strict_capacity_limit : 0 2024-04-16T05:45:19.318 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: high_pri_pool_ratio: 0.000 2024-04-16T05:45:19.318 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_cache_compressed: (nil) 2024-04-16T05:45:19.318 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: persistent_cache: (nil) 2024-04-16T05:45:19.318 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_size: 4096 2024-04-16T05:45:19.318 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_size_deviation: 10 2024-04-16T05:45:19.318 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_restart_interval: 16 2024-04-16T05:45:19.318 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: index_block_restart_interval: 1 2024-04-16T05:45:19.319 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: metadata_block_size: 4096 2024-04-16T05:45:19.319 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: partition_filters: 0 2024-04-16T05:45:19.319 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: use_delta_encoding: 1 2024-04-16T05:45:19.319 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: filter_policy: bloomfilter 2024-04-16T05:45:19.319 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: whole_key_filtering: 1 2024-04-16T05:45:19.319 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: verify_compression: 0 2024-04-16T05:45:19.319 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: read_amp_bytes_per_bit: 0 2024-04-16T05:45:19.319 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: format_version: 5 2024-04-16T05:45:19.319 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: enable_index_compression: 1 2024-04-16T05:45:19.319 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: block_align: 0 2024-04-16T05:45:19.319 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: max_auto_readahead_size: 262144 2024-04-16T05:45:19.319 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: prepopulate_block_cache: 0 2024-04-16T05:45:19.319 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: initial_auto_readahead_size: 8192 2024-04-16T05:45:19.319 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: num_file_reads_for_auto_readahead: 2 2024-04-16T05:45:19.319 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.write_buffer_size: 33554432 2024-04-16T05:45:19.320 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_write_buffer_number: 2 2024-04-16T05:45:19.320 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compression: NoCompression 2024-04-16T05:45:19.320 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.bottommost_compression: Disabled 2024-04-16T05:45:19.320 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.prefix_extractor: nullptr 2024-04-16T05:45:19.320 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-16T05:45:19.320 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.num_levels: 7 2024-04-16T05:45:19.320 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-16T05:45:19.320 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-16T05:45:19.320 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-16T05:45:19.320 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-16T05:45:19.320 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-16T05:45:19.320 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-16T05:45:19.320 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-16T05:45:19.320 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-16T05:45:19.320 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-16T05:45:19.320 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-16T05:45:19.320 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-16T05:45:19.321 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-16T05:45:19.321 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-16T05:45:19.321 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compression_opts.level: 32767 2024-04-16T05:45:19.321 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compression_opts.strategy: 0 2024-04-16T05:45:19.321 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-16T05:45:19.321 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-16T05:45:19.321 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-16T05:45:19.321 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-16T05:45:19.321 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compression_opts.enabled: false 2024-04-16T05:45:19.321 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-16T05:45:19.321 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-16T05:45:19.321 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-16T05:45:19.321 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-16T05:45:19.321 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.target_file_size_base: 67108864 2024-04-16T05:45:19.321 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-16T05:45:19.321 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-16T05:45:19.321 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-16T05:45:19.322 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-16T05:45:19.322 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-16T05:45:19.323 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-16T05:45:19.323 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-16T05:45:19.323 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-16T05:45:19.323 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-16T05:45:19.323 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-16T05:45:19.323 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-16T05:45:19.323 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-16T05:45:19.323 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-16T05:45:19.323 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-16T05:45:19.323 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.arena_block_size: 1048576 2024-04-16T05:45:19.323 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-16T05:45:19.323 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-16T05:45:19.323 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.disable_auto_compactions: 0 2024-04-16T05:45:19.323 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-16T05:45:19.323 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-16T05:45:19.324 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-16T05:45:19.324 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-16T05:45:19.324 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-16T05:45:19.324 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-16T05:45:19.324 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-16T05:45:19.324 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-16T05:45:19.324 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-16T05:45:19.324 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-16T05:45:19.324 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-16T05:45:19.324 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.inplace_update_support: 0 2024-04-16T05:45:19.324 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-16T05:45:19.324 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-16T05:45:19.324 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-16T05:45:19.324 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-16T05:45:19.324 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.bloom_locality: 0 2024-04-16T05:45:19.324 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.max_successive_merges: 0 2024-04-16T05:45:19.325 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-16T05:45:19.325 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.paranoid_file_checks: 0 2024-04-16T05:45:19.325 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.force_consistency_checks: 1 2024-04-16T05:45:19.325 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.report_bg_io_stats: 0 2024-04-16T05:45:19.325 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.ttl: 2592000 2024-04-16T05:45:19.325 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-16T05:45:19.325 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-16T05:45:19.325 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-16T05:45:19.325 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.enable_blob_files: false 2024-04-16T05:45:19.325 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.min_blob_size: 0 2024-04-16T05:45:19.325 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.blob_file_size: 268435456 2024-04-16T05:45:19.325 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-16T05:45:19.325 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-16T05:45:19.325 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-16T05:45:19.325 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-16T05:45:19.325 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-16T05:45:19.326 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.blob_file_starting_level: 0 2024-04-16T05:45:19.326 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-16T05:45:19.326 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 33.sst 2024-04-16T05:45:19.326 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-16T05:45:19.326 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi027/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 35, last_sequence is 11773, log_number is 31,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-16T05:45:19.326 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 31 2024-04-16T05:45:19.326 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 08c26cc4-d879-4639-87a0-7a5f9a4f6d9e 2024-04-16T05:45:19.326 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713246319120980, "job": 1, "event": "recovery_started", "wal_files": [31]} 2024-04-16T05:45:19.326 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #31 mode 2 2024-04-16T05:45:19.326 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713246319131805, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 36, "file_size": 1976613, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11775, "largest_seqno": 12687, "table_properties": {"data_size": 1971622, "index_size": 2862, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1093, "raw_key_size": 9580, "raw_average_key_size": 23, "raw_value_size": 1962568, "raw_average_value_size": 4751, "num_data_blocks": 131, "num_entries": 413, "num_filter_entries": 413, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713246319, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "08c26cc4-d879-4639-87a0-7a5f9a4f6d9e", "db_session_id": "027YB5KU1BYLDJQGSCIX", "orig_file_number": 36, "seqno_to_time_mapping": "N/A"}} 2024-04-16T05:45:19.326 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713246319131988, "job": 1, "event": "recovery_finished"} 2024-04-16T05:45:19.326 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: [db/version_set.cc:5047] Creating manifest 38 2024-04-16T05:45:19.326 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-16T05:45:19.326 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi027/store.db/000031.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-16T05:45:19.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5564c4e86000 2024-04-16T05:45:19.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: DB pointer 0x5564c4e70000 2024-04-16T05:45:19.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-16T05:45:19.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-16T05:45:19.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ** DB Stats ** 2024-04-16T05:45:19.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T05:45:19.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-16T05:45:19.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T05:45:19.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T05:45:19.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-16T05:45:19.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T05:45:19.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T05:45:19.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-16T05:45:19.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ** Compaction Stats [default] ** 2024-04-16T05:45:19.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-16T05:45:19.328 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-16T05:45:19.328 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: L0 1/0 1.89 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 387.6 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-16T05:45:19.328 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: L6 1/0 5.48 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-16T05:45:19.328 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Sum 2/0 7.37 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 387.6 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-16T05:45:19.328 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 387.6 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-16T05:45:19.328 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-16T05:45:19.328 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ** Compaction Stats [default] ** 2024-04-16T05:45:19.328 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-16T05:45:19.328 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T05:45:19.328 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 387.6 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-04-16T05:45:19.328 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-16T05:45:19.328 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-16T05:45:19.328 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-16T05:45:19.328 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T05:45:19.328 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-04-16T05:45:19.328 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-16T05:45:19.329 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-16T05:45:19.329 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-16T05:45:19.329 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-16T05:45:19.329 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Cumulative compaction: 0.00 GB write, 108.80 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T05:45:19.329 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Interval compaction: 0.00 GB write, 108.80 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T05:45:19.329 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-16T05:45:19.329 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Block cache BinnedLRUCache@0x5564c4d97090#2 capacity: 512.00 MB usage: 4.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.5e-05 secs_since: 0 2024-04-16T05:45:19.329 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.16 KB,0.000220537%) IndexBlock(1,3.08 KB,0.000587106%) Misc(1,0.00 KB,0%) 2024-04-16T05:45:19.329 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-16T05:45:19.329 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-16T05:45:19.329 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: starting mon.smithi027 rank 0 at public addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] at bind addrs [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi027 fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:45:19.329 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: mon.smithi027@-1(???) e2 preinit fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:45:19.329 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: mon.smithi027@-1(???).mds e16 new map 2024-04-16T05:45:19.329 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: mon.smithi027@-1(???).mds e16 print_map 2024-04-16T05:45:19.329 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: e16 2024-04-16T05:45:19.330 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:45:19.330 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:45:19.330 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: legacy client fscid: 1 2024-04-16T05:45:19.330 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-16T05:45:19.330 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Filesystem 'cephfs' (1) 2024-04-16T05:45:19.330 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: fs_name cephfs 2024-04-16T05:45:19.330 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: epoch 15 2024-04-16T05:45:19.330 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:45:19.330 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:45:19.330 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: modified 2024-04-16T05:40:32.701649+0000 2024-04-16T05:45:19.330 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: tableserver 0 2024-04-16T05:45:19.330 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: root 0 2024-04-16T05:45:19.330 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: session_timeout 60 2024-04-16T05:45:19.330 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: session_autoclose 300 2024-04-16T05:45:19.330 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: max_file_size 1099511627776 2024-04-16T05:45:19.331 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: max_xattr_size 65536 2024-04-16T05:45:19.331 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: required_client_features {} 2024-04-16T05:45:19.331 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: last_failure 0 2024-04-16T05:45:19.331 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: last_failure_osd_epoch 46 2024-04-16T05:45:19.331 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:45:19.331 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: max_mds 1 2024-04-16T05:45:19.331 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: in 0 2024-04-16T05:45:19.331 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: up {0=14462} 2024-04-16T05:45:19.331 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: failed 2024-04-16T05:45:19.331 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: damaged 2024-04-16T05:45:19.331 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: stopped 2024-04-16T05:45:19.331 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: data_pools [3] 2024-04-16T05:45:19.331 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: metadata_pool 2 2024-04-16T05:45:19.332 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: inline_data disabled 2024-04-16T05:45:19.332 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: balancer 2024-04-16T05:45:19.332 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: bal_rank_mask -1 2024-04-16T05:45:19.332 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: standby_count_wanted 1 2024-04-16T05:45:19.332 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: qdb_cluster leader: 0 members: 2024-04-16T05:45:19.332 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: [mds.cephfs.smithi027.ejoytk{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:45:19.332 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-16T05:45:19.332 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-16T05:45:19.332 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: Standby daemons: 2024-04-16T05:45:19.332 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: 2024-04-16T05:45:19.332 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: [mds.cephfs.smithi027.hkhpuq{-1:14474} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2065690722,v1:172.21.15.27:6827/2065690722] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:45:19.332 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: [mds.cephfs.smithi122.lojjdp{-1:24263} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:45:19.332 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout: [mds.cephfs.smithi122.rulewi{-1:24273} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:45:19.332 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: mon.smithi027@-1(???).osd e48 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-16T05:45:19.332 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: mon.smithi027@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T05:45:19.332 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: mon.smithi027@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T05:45:19.332 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: mon.smithi027@-1(???).osd e48 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T05:45:19.332 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: mon.smithi027@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-04-16T05:45:19.333 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 bash[93617]: 299ca337e4ca2f01879e71aeb313a8b644c35532d69e0163715bc02d510b84ba 2024-04-16T05:45:19.333 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 systemd[1]: Started Ceph mon.smithi027 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968. 2024-04-16T05:45:19.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:19 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-16T05:45:19.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:19 smithi122 ceph-mon[35998]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:45:19.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:19 smithi122 ceph-mon[35998]: pgmap v30: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:19.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:19 smithi122 ceph-mon[35998]: mon.smithi027 calling monitor election 2024-04-16T05:45:19.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:19 smithi122 ceph-mon[35998]: mon.smithi027 is new leader, mons smithi027,smithi122 in quorum (ranks 0,1) 2024-04-16T05:45:19.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:19 smithi122 ceph-mon[35998]: monmap epoch 2 2024-04-16T05:45:19.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:19 smithi122 ceph-mon[35998]: fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:45:19.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:19 smithi122 ceph-mon[35998]: last_changed 2024-04-16T05:34:24.745534+0000 2024-04-16T05:45:19.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:19 smithi122 ceph-mon[35998]: created 2024-04-16T05:30:56.860193+0000 2024-04-16T05:45:19.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:19 smithi122 ceph-mon[35998]: min_mon_release 17 (quincy) 2024-04-16T05:45:19.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:19 smithi122 ceph-mon[35998]: election_strategy: 1 2024-04-16T05:45:19.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:19 smithi122 ceph-mon[35998]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.smithi027 2024-04-16T05:45:19.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:19 smithi122 ceph-mon[35998]: 1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.smithi122 2024-04-16T05:45:19.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:19 smithi122 ceph-mon[35998]: fsmap cephfs:1 {0=cephfs.smithi027.ejoytk=up:active} 3 up:standby 2024-04-16T05:45:19.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:19 smithi122 ceph-mon[35998]: osdmap e48: 6 total, 6 up, 6 in 2024-04-16T05:45:19.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:19 smithi122 ceph-mon[35998]: mgrmap e32: smithi027.qgzypi(active, since 55s), standbys: smithi122.sgokkq 2024-04-16T05:45:19.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:19 smithi122 ceph-mon[35998]: overall HEALTH_OK 2024-04-16T05:45:19.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:19 smithi122 ceph-mon[35998]: from='mgr.14716 ' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:19.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:19 smithi122 ceph-mon[35998]: mgrmap e33: smithi027.qgzypi(active, since 55s), standbys: smithi122.sgokkq 2024-04-16T05:45:19.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:19 smithi122 ceph-mon[35998]: from='mgr.14716 ' entity='' 2024-04-16T05:45:19.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-16T05:45:19.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: from='mgr.14716 172.21.15.27:0/1483264801' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:45:19.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: pgmap v30: 65 pgs: 65 active+clean; 582 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:19.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: mon.smithi027 calling monitor election 2024-04-16T05:45:19.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: mon.smithi027 is new leader, mons smithi027,smithi122 in quorum (ranks 0,1) 2024-04-16T05:45:19.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: monmap epoch 2 2024-04-16T05:45:19.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:45:19.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: last_changed 2024-04-16T05:34:24.745534+0000 2024-04-16T05:45:19.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: created 2024-04-16T05:30:56.860193+0000 2024-04-16T05:45:19.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: min_mon_release 17 (quincy) 2024-04-16T05:45:19.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: election_strategy: 1 2024-04-16T05:45:19.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.smithi027 2024-04-16T05:45:19.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: 1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.smithi122 2024-04-16T05:45:19.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: fsmap cephfs:1 {0=cephfs.smithi027.ejoytk=up:active} 3 up:standby 2024-04-16T05:45:19.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: osdmap e48: 6 total, 6 up, 6 in 2024-04-16T05:45:19.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: mgrmap e32: smithi027.qgzypi(active, since 55s), standbys: smithi122.sgokkq 2024-04-16T05:45:19.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: overall HEALTH_OK 2024-04-16T05:45:19.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: from='mgr.14716 ' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:19.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: mgrmap e33: smithi027.qgzypi(active, since 55s), standbys: smithi122.sgokkq 2024-04-16T05:45:19.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:19 smithi027 ceph-mon[93635]: from='mgr.14716 ' entity='' 2024-04-16T05:45:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.472+0000 7f9817000700 1 -- 172.21.15.27:0/3215472286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98100fcef0 msgr2=0x7f98100fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.472+0000 7f9817000700 1 --2- 172.21.15.27:0/3215472286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98100fcef0 0x7f98100fd320 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9800004940 tx=0x7f9800009e30 comp rx=0 tx=0).stop 2024-04-16T05:45:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.472+0000 7f9817000700 1 -- 172.21.15.27:0/3215472286 shutdown_connections 2024-04-16T05:45:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.472+0000 7f9817000700 1 --2- 172.21.15.27:0/3215472286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98100fe0e0 0x7f98100fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.472+0000 7f9817000700 1 --2- 172.21.15.27:0/3215472286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98100fcef0 0x7f98100fd320 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.472+0000 7f9817000700 1 -- 172.21.15.27:0/3215472286 >> 172.21.15.27:0/3215472286 conn(0x7f98100f8530 msgr2=0x7f98100fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:45:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.473+0000 7f9817000700 1 -- 172.21.15.27:0/3215472286 shutdown_connections 2024-04-16T05:45:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.473+0000 7f9817000700 1 -- 172.21.15.27:0/3215472286 wait complete. 2024-04-16T05:45:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.473+0000 7f9817000700 1 Processor -- start 2024-04-16T05:45:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.474+0000 7f9817000700 1 -- start start 2024-04-16T05:45:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.474+0000 7f9817000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98100fcef0 0x7f9810071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.474+0000 7f9817000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98100fe0e0 0x7f9810070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:23.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.474+0000 7f9817000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98100708c0 con 0x7f98100fcef0 2024-04-16T05:45:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.474+0000 7f9817000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9810070a30 con 0x7f98100fe0e0 2024-04-16T05:45:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.474+0000 7f980fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98100fcef0 0x7f9810071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:45:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.474+0000 7f980fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98100fcef0 0x7f9810071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38460/0 (socket says 172.21.15.27:38460) 2024-04-16T05:45:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.474+0000 7f980fe00700 1 -- 172.21.15.27:0/297637154 learned_addr learned my addr 172.21.15.27:0/297637154 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:45:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.475+0000 7f980f400700 1 --2- 172.21.15.27:0/297637154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98100fe0e0 0x7f9810070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:45:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.475+0000 7f980fe00700 1 -- 172.21.15.27:0/297637154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98100fe0e0 msgr2=0x7f9810070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.475+0000 7f980fe00700 1 --2- 172.21.15.27:0/297637154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98100fe0e0 0x7f9810070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:23.474 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.475+0000 7f980fe00700 1 -- 172.21.15.27:0/297637154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9800005040 con 0x7f98100fcef0 2024-04-16T05:45:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.475+0000 7f980fe00700 1 --2- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98100fcef0 0x7f9810071e40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f980000b970 tx=0x7f9800009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:45:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.475+0000 7f9807e00700 1 -- 172.21.15.27:0/297637154 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f980000e040 con 0x7f98100fcef0 2024-04-16T05:45:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.476+0000 7f9807e00700 1 -- 172.21.15.27:0/297637154 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9800004290 con 0x7f98100fcef0 2024-04-16T05:45:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.476+0000 7f9817000700 1 -- 172.21.15.27:0/297637154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9810070cb0 con 0x7f98100fcef0 2024-04-16T05:45:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.476+0000 7f9807e00700 1 -- 172.21.15.27:0/297637154 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f980001c630 con 0x7f98100fcef0 2024-04-16T05:45:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.476+0000 7f9817000700 1 -- 172.21.15.27:0/297637154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9810115ab0 con 0x7f98100fcef0 2024-04-16T05:45:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.477+0000 7f9817000700 1 -- 172.21.15.27:0/297637154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f981005f5f0 con 0x7f98100fcef0 2024-04-16T05:45:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.478+0000 7f9807e00700 1 -- 172.21.15.27:0/297637154 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 33) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f980000bbf0 con 0x7f98100fcef0 2024-04-16T05:45:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.479+0000 7f9807e00700 1 --2- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f97f406ed20 0x7f97f40711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.479+0000 7f980f400700 1 -- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f97f406ed20 msgr2=0x7f97f40711d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1090905394 2024-04-16T05:45:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.479+0000 7f9807e00700 1 -- 172.21.15.27:0/297637154 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6152+0+0 (secure 0 0 0) 0x7f980008fdf0 con 0x7f98100fcef0 2024-04-16T05:45:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.479+0000 7f980f400700 1 --2- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f97f406ed20 0x7f97f40711d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-16T05:45:23.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.491+0000 7f9807e00700 1 -- 172.21.15.27:0/297637154 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9800058ab0 con 0x7f98100fcef0 2024-04-16T05:45:23.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.680+0000 7f980f400700 1 -- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f97f406ed20 msgr2=0x7f97f40711d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1090905394 2024-04-16T05:45:23.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.680+0000 7f980f400700 1 --2- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f97f406ed20 0x7f97f40711d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-16T05:45:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:23.739+0000 7f9817000700 1 -- 172.21.15.27:0/297637154 --> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9810102440 con 0x7f97f406ed20 2024-04-16T05:45:24.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:24.081+0000 7f980f400700 1 -- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f97f406ed20 msgr2=0x7f97f40711d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1090905394 2024-04-16T05:45:24.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:24.081+0000 7f980f400700 1 --2- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f97f406ed20 0x7f97f40711d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-16T05:45:24.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:24.882+0000 7f980f400700 1 -- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f97f406ed20 msgr2=0x7f97f40711d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1090905394 2024-04-16T05:45:24.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:24.882+0000 7f980f400700 1 --2- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f97f406ed20 0x7f97f40711d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-16T05:45:26.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:26.484+0000 7f980f400700 1 -- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f97f406ed20 msgr2=0x7f97f40711d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.27:6800/1090905394 2024-04-16T05:45:26.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:26.484+0000 7f980f400700 1 --2- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f97f406ed20 0x7f97f40711d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-16T05:45:28.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:28.466+0000 7f9807e00700 1 -- 172.21.15.27:0/297637154 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 34) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f980008e0a0 con 0x7f98100fcef0 2024-04-16T05:45:28.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:28 smithi122 ceph-mon[35998]: Standby manager daemon smithi122.sgokkq restarted 2024-04-16T05:45:28.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:28 smithi122 ceph-mon[35998]: Standby manager daemon smithi122.sgokkq started 2024-04-16T05:45:28.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:28 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.122:0/3817665728' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.sgokkq/crt"}]: dispatch 2024-04-16T05:45:28.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:28 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.122:0/3817665728' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T05:45:28.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:28 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.122:0/3817665728' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.sgokkq/key"}]: dispatch 2024-04-16T05:45:28.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:28 smithi122 ceph-mon[35998]: from='mgr.? 172.21.15.122:0/3817665728' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T05:45:28.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:28 smithi027 ceph-mon[93635]: Standby manager daemon smithi122.sgokkq restarted 2024-04-16T05:45:28.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:28 smithi027 ceph-mon[93635]: Standby manager daemon smithi122.sgokkq started 2024-04-16T05:45:28.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:28 smithi027 ceph-mon[93635]: from='mgr.? 172.21.15.122:0/3817665728' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.sgokkq/crt"}]: dispatch 2024-04-16T05:45:28.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:28 smithi027 ceph-mon[93635]: from='mgr.? 172.21.15.122:0/3817665728' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-16T05:45:28.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:28 smithi027 ceph-mon[93635]: from='mgr.? 172.21.15.122:0/3817665728' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi122.sgokkq/key"}]: dispatch 2024-04-16T05:45:28.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:28 smithi027 ceph-mon[93635]: from='mgr.? 172.21.15.122:0/3817665728' entity='mgr.smithi122.sgokkq' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-16T05:45:28.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:28.886+0000 7f9807e00700 1 -- 172.21.15.27:0/297637154 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mgrmap(e 35) v1 ==== 92370+0+0 (secure 0 0 0) 0x7f9800058e80 con 0x7f98100fcef0 2024-04-16T05:45:28.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:28.886+0000 7f9807e00700 1 -- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f97f406ed20 msgr2=0x7f97f40711d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-16T05:45:28.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:28.886+0000 7f9807e00700 1 --2- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f97f406ed20 0x7f97f40711d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:29.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: mgrmap e34: smithi027.qgzypi(active, since 64s), standbys: smithi122.sgokkq 2024-04-16T05:45:29.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: Active manager daemon smithi027.qgzypi restarted 2024-04-16T05:45:29.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: Activating manager daemon smithi027.qgzypi 2024-04-16T05:45:29.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: osdmap e49: 6 total, 6 up, 6 in 2024-04-16T05:45:29.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: mgrmap e35: smithi027.qgzypi(active, starting, since 0.0179021s), standbys: smithi122.sgokkq 2024-04-16T05:45:29.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-16T05:45:29.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:45:29.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.ejoytk"}]: dispatch 2024-04-16T05:45:29.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.hkhpuq"}]: dispatch 2024-04-16T05:45:29.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.lojjdp"}]: dispatch 2024-04-16T05:45:29.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.rulewi"}]: dispatch 2024-04-16T05:45:29.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr metadata", "who": "smithi027.qgzypi", "id": "smithi027.qgzypi"}]: dispatch 2024-04-16T05:45:29.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr metadata", "who": "smithi122.sgokkq", "id": "smithi122.sgokkq"}]: dispatch 2024-04-16T05:45:29.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T05:45:29.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T05:45:29.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T05:45:29.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T05:45:29.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T05:45:29.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T05:45:29.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T05:45:29.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T05:45:29.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T05:45:29.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: Manager daemon smithi027.qgzypi is now available 2024-04-16T05:45:29.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:45:29.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:45:29.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.qgzypi/mirror_snapshot_schedule"}]: dispatch 2024-04-16T05:45:29.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:29 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.qgzypi/trash_purge_schedule"}]: dispatch 2024-04-16T05:45:29.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: mgrmap e34: smithi027.qgzypi(active, since 64s), standbys: smithi122.sgokkq 2024-04-16T05:45:29.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: Active manager daemon smithi027.qgzypi restarted 2024-04-16T05:45:29.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: Activating manager daemon smithi027.qgzypi 2024-04-16T05:45:29.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: osdmap e49: 6 total, 6 up, 6 in 2024-04-16T05:45:29.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: mgrmap e35: smithi027.qgzypi(active, starting, since 0.0179021s), standbys: smithi122.sgokkq 2024-04-16T05:45:29.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-16T05:45:29.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:45:29.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.ejoytk"}]: dispatch 2024-04-16T05:45:29.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.hkhpuq"}]: dispatch 2024-04-16T05:45:29.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.lojjdp"}]: dispatch 2024-04-16T05:45:29.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.rulewi"}]: dispatch 2024-04-16T05:45:29.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr metadata", "who": "smithi027.qgzypi", "id": "smithi027.qgzypi"}]: dispatch 2024-04-16T05:45:29.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr metadata", "who": "smithi122.sgokkq", "id": "smithi122.sgokkq"}]: dispatch 2024-04-16T05:45:29.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T05:45:29.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T05:45:29.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T05:45:29.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T05:45:29.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T05:45:29.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T05:45:29.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-16T05:45:29.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-16T05:45:29.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-16T05:45:29.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: Manager daemon smithi027.qgzypi is now available 2024-04-16T05:45:29.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:45:29.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:45:29.810 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.qgzypi/mirror_snapshot_schedule"}]: dispatch 2024-04-16T05:45:29.810 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi027.qgzypi/trash_purge_schedule"}]: dispatch 2024-04-16T05:45:29.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:29.893+0000 7f9807e00700 1 -- 172.21.15.27:0/297637154 <== mon.0 v2:172.21.15.27:3300/0 9 ==== mgrmap(e 36) v1 ==== 92497+0+0 (secure 0 0 0) 0x7f9800059750 con 0x7f98100fcef0 2024-04-16T05:45:29.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:29.894+0000 7f9807e00700 1 --2- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f97f4029440 0x7f97f4029810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:29.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:29.894+0000 7f9807e00700 1 -- 172.21.15.27:0/297637154 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9810102440 con 0x7f97f4029440 2024-04-16T05:45:29.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:29.898+0000 7f980f400700 1 --2- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f97f4029440 0x7f97f4029810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:45:29.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:29.900+0000 7f980f400700 1 --2- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f97f4029440 0x7f97f4029810 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0041d0 tx=0x7f97fc009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:45:29.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:29.906+0000 7f9807e00700 1 -- 172.21.15.27:0/297637154 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f9810102440 con 0x7f97f4029440 2024-04-16T05:45:29.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:29.912+0000 7f9817000700 1 -- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f97f4029440 msgr2=0x7f97f4029810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:29.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:29.912+0000 7f9817000700 1 --2- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f97f4029440 0x7f97f4029810 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f97fc0041d0 tx=0x7f97fc009040 comp rx=0 tx=0).stop 2024-04-16T05:45:29.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:29.912+0000 7f9817000700 1 -- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98100fcef0 msgr2=0x7f9810071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:29.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:29.912+0000 7f9817000700 1 --2- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98100fcef0 0x7f9810071e40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f980000b970 tx=0x7f9800009e30 comp rx=0 tx=0).stop 2024-04-16T05:45:29.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:29.912+0000 7f9817000700 1 -- 172.21.15.27:0/297637154 shutdown_connections 2024-04-16T05:45:29.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:29.912+0000 7f9817000700 1 --2- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f97f4029440 0x7f97f4029810 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:29.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:29.912+0000 7f9817000700 1 --2- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f98100fcef0 0x7f9810071e40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:29.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:29.912+0000 7f9817000700 1 --2- 172.21.15.27:0/297637154 >> [v2:172.21.15.27:6800/1090905394,v1:172.21.15.27:6801/1090905394] conn(0x7f97f406ed20 0x7f97f40711d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:29.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:29.912+0000 7f9817000700 1 --2- 172.21.15.27:0/297637154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f98100fe0e0 0x7f9810070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:29.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:29.912+0000 7f9817000700 1 -- 172.21.15.27:0/297637154 >> 172.21.15.27:0/297637154 conn(0x7f98100f8530 msgr2=0x7f98100f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:45:29.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:29.913+0000 7f9817000700 1 -- 172.21.15.27:0/297637154 shutdown_connections 2024-04-16T05:45:29.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:29.913+0000 7f9817000700 1 -- 172.21.15.27:0/297637154 wait complete. 2024-04-16T05:45:29.926 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:45:30.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.070+0000 7f68edc00700 1 -- 172.21.15.27:0/1579046391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68e80709a0 msgr2=0x7f68e8070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:30.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.070+0000 7f68edc00700 1 --2- 172.21.15.27:0/1579046391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68e80709a0 0x7f68e8070e10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f68d4004940 tx=0x7f68d4009e30 comp rx=0 tx=0).stop 2024-04-16T05:45:30.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.070+0000 7f68edc00700 1 -- 172.21.15.27:0/1579046391 shutdown_connections 2024-04-16T05:45:30.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.070+0000 7f68edc00700 1 --2- 172.21.15.27:0/1579046391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68e80709a0 0x7f68e8070e10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:30.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.070+0000 7f68edc00700 1 --2- 172.21.15.27:0/1579046391 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68e8071f50 0x7f68e8070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:30.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.070+0000 7f68edc00700 1 -- 172.21.15.27:0/1579046391 >> 172.21.15.27:0/1579046391 conn(0x7f68e80f8530 msgr2=0x7f68e80fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:45:30.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.070+0000 7f68edc00700 1 -- 172.21.15.27:0/1579046391 shutdown_connections 2024-04-16T05:45:30.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.070+0000 7f68edc00700 1 -- 172.21.15.27:0/1579046391 wait complete. 2024-04-16T05:45:30.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.071+0000 7f68edc00700 1 Processor -- start 2024-04-16T05:45:30.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.072+0000 7f68edc00700 1 -- start start 2024-04-16T05:45:30.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.072+0000 7f68edc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68e80709a0 0x7f68e810fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:30.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.072+0000 7f68edc00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68e8071f50 0x7f68e8110100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:30.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.072+0000 7f68edc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e8110700 con 0x7f68e80709a0 2024-04-16T05:45:30.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.072+0000 7f68edc00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e8110f50 con 0x7f68e8071f50 2024-04-16T05:45:30.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.072+0000 7f68e6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68e80709a0 0x7f68e810fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:45:30.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.072+0000 7f68e6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68e80709a0 0x7f68e810fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41290/0 (socket says 172.21.15.27:41290) 2024-04-16T05:45:30.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.072+0000 7f68e6a00700 1 -- 172.21.15.27:0/1596057436 learned_addr learned my addr 172.21.15.27:0/1596057436 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:45:30.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.072+0000 7f68e6000700 1 --2- 172.21.15.27:0/1596057436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68e8071f50 0x7f68e8110100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:45:30.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.073+0000 7f68e6a00700 1 -- 172.21.15.27:0/1596057436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68e8071f50 msgr2=0x7f68e8110100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:30.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.073+0000 7f68e6a00700 1 --2- 172.21.15.27:0/1596057436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68e8071f50 0x7f68e8110100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:30.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.073+0000 7f68e6a00700 1 -- 172.21.15.27:0/1596057436 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d4005040 con 0x7f68e80709a0 2024-04-16T05:45:30.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.073+0000 7f68e6a00700 1 --2- 172.21.15.27:0/1596057436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68e80709a0 0x7f68e810fbc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f68d800eb70 tx=0x7f68d800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:45:30.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.073+0000 7f68dea00700 1 -- 172.21.15.27:0/1596057436 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68d8007cc0 con 0x7f68e80709a0 2024-04-16T05:45:30.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.073+0000 7f68dea00700 1 -- 172.21.15.27:0/1596057436 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f68d800f040 con 0x7f68e80709a0 2024-04-16T05:45:30.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.073+0000 7f68edc00700 1 -- 172.21.15.27:0/1596057436 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e81111d0 con 0x7f68e80709a0 2024-04-16T05:45:30.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.073+0000 7f68dea00700 1 -- 172.21.15.27:0/1596057436 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68d800cbd0 con 0x7f68e80709a0 2024-04-16T05:45:30.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.074+0000 7f68edc00700 1 -- 172.21.15.27:0/1596057436 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e8111710 con 0x7f68e80709a0 2024-04-16T05:45:30.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.075+0000 7f68edc00700 1 -- 172.21.15.27:0/1596057436 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e805f5f0 con 0x7f68e80709a0 2024-04-16T05:45:30.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.076+0000 7f68dea00700 1 -- 172.21.15.27:0/1596057436 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 36) v1 ==== 92497+0+0 (secure 0 0 0) 0x7f68d800cd30 con 0x7f68e80709a0 2024-04-16T05:45:30.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.077+0000 7f68dea00700 1 --2- 172.21.15.27:0/1596057436 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f68cc06e900 0x7f68cc070db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:30.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.077+0000 7f68dea00700 1 -- 172.21.15.27:0/1596057436 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f68d808de70 con 0x7f68e80709a0 2024-04-16T05:45:30.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.077+0000 7f68e6000700 1 --2- 172.21.15.27:0/1596057436 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f68cc06e900 0x7f68cc070db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:45:30.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.078+0000 7f68e6000700 1 --2- 172.21.15.27:0/1596057436 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f68cc06e900 0x7f68cc070db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f68d400b0d0 tx=0x7f68d4009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:45:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.085+0000 7f68dea00700 1 -- 172.21.15.27:0/1596057436 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f68d8056f50 con 0x7f68e80709a0 2024-04-16T05:45:30.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.340+0000 7f68edc00700 1 -- 172.21.15.27:0/1596057436 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f68e8111a40 con 0x7f68cc06e900 2024-04-16T05:45:30.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.343+0000 7f68dea00700 1 -- 172.21.15.27:0/1596057436 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7f68e8111a40 con 0x7f68cc06e900 2024-04-16T05:45:30.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.345+0000 7f68edc00700 1 -- 172.21.15.27:0/1596057436 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f68cc06e900 msgr2=0x7f68cc070db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:30.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.345+0000 7f68edc00700 1 --2- 172.21.15.27:0/1596057436 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f68cc06e900 0x7f68cc070db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f68d400b0d0 tx=0x7f68d4009e30 comp rx=0 tx=0).stop 2024-04-16T05:45:30.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.345+0000 7f68edc00700 1 -- 172.21.15.27:0/1596057436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68e80709a0 msgr2=0x7f68e810fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:30.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.345+0000 7f68edc00700 1 --2- 172.21.15.27:0/1596057436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68e80709a0 0x7f68e810fbc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f68d800eb70 tx=0x7f68d800b040 comp rx=0 tx=0).stop 2024-04-16T05:45:30.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.346+0000 7f68edc00700 1 -- 172.21.15.27:0/1596057436 shutdown_connections 2024-04-16T05:45:30.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.346+0000 7f68edc00700 1 --2- 172.21.15.27:0/1596057436 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f68cc06e900 0x7f68cc070db0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:30.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.346+0000 7f68edc00700 1 --2- 172.21.15.27:0/1596057436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f68e80709a0 0x7f68e810fbc0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:30.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.346+0000 7f68edc00700 1 --2- 172.21.15.27:0/1596057436 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68e8071f50 0x7f68e8110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:30.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.346+0000 7f68edc00700 1 -- 172.21.15.27:0/1596057436 >> 172.21.15.27:0/1596057436 conn(0x7f68e80f8530 msgr2=0x7f68e80f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:45:30.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.346+0000 7f68edc00700 1 -- 172.21.15.27:0/1596057436 shutdown_connections 2024-04-16T05:45:30.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.346+0000 7f68edc00700 1 -- 172.21.15.27:0/1596057436 wait complete. 2024-04-16T05:45:30.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.491+0000 7f359da00700 1 -- 172.21.15.27:0/4073606371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3598103420 msgr2=0x7f3598103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:30.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.491+0000 7f359da00700 1 --2- 172.21.15.27:0/4073606371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3598103420 0x7f3598103830 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3588004940 tx=0x7f3588009e30 comp rx=0 tx=0).stop 2024-04-16T05:45:30.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.491+0000 7f359da00700 1 -- 172.21.15.27:0/4073606371 shutdown_connections 2024-04-16T05:45:30.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.491+0000 7f359da00700 1 --2- 172.21.15.27:0/4073606371 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35981046d0 0x7f3598104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:30.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.491+0000 7f359da00700 1 --2- 172.21.15.27:0/4073606371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3598103420 0x7f3598103830 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:30.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.491+0000 7f359da00700 1 -- 172.21.15.27:0/4073606371 >> 172.21.15.27:0/4073606371 conn(0x7f3598072640 msgr2=0x7f3598074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:45:30.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.491+0000 7f359da00700 1 -- 172.21.15.27:0/4073606371 shutdown_connections 2024-04-16T05:45:30.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.492+0000 7f359da00700 1 -- 172.21.15.27:0/4073606371 wait complete. 2024-04-16T05:45:30.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.492+0000 7f359da00700 1 Processor -- start 2024-04-16T05:45:30.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.493+0000 7f359da00700 1 -- start start 2024-04-16T05:45:30.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.493+0000 7f359da00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3598103420 0x7f3598113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:30.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.493+0000 7f359da00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35981046d0 0x7f35981144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:30.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.493+0000 7f359da00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3598114aa0 con 0x7f35981046d0 2024-04-16T05:45:30.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.493+0000 7f359da00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3598114c10 con 0x7f3598103420 2024-04-16T05:45:30.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.493+0000 7f3596000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35981046d0 0x7f35981144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:45:30.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.493+0000 7f3596000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35981046d0 0x7f35981144a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41306/0 (socket says 172.21.15.27:41306) 2024-04-16T05:45:30.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.493+0000 7f3596000700 1 -- 172.21.15.27:0/2503875327 learned_addr learned my addr 172.21.15.27:0/2503875327 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:45:30.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.494+0000 7f3596a00700 1 --2- 172.21.15.27:0/2503875327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3598103420 0x7f3598113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:45:30.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.494+0000 7f3596000700 1 -- 172.21.15.27:0/2503875327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3598103420 msgr2=0x7f3598113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:30.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.494+0000 7f3596000700 1 --2- 172.21.15.27:0/2503875327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3598103420 0x7f3598113f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:30.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.494+0000 7f3596000700 1 -- 172.21.15.27:0/2503875327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3588005040 con 0x7f35981046d0 2024-04-16T05:45:30.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.494+0000 7f3596000700 1 --2- 172.21.15.27:0/2503875327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35981046d0 0x7f35981144a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f358400c8c0 tx=0x7f35840044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:45:30.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.494+0000 7f358ea00700 1 -- 172.21.15.27:0/2503875327 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f358400a980 con 0x7f35981046d0 2024-04-16T05:45:30.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.494+0000 7f358ea00700 1 -- 172.21.15.27:0/2503875327 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f358400ce60 con 0x7f35981046d0 2024-04-16T05:45:30.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.494+0000 7f358ea00700 1 -- 172.21.15.27:0/2503875327 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35840115e0 con 0x7f35981046d0 2024-04-16T05:45:30.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.494+0000 7f359da00700 1 -- 172.21.15.27:0/2503875327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3598119670 con 0x7f35981046d0 2024-04-16T05:45:30.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.495+0000 7f359da00700 1 -- 172.21.15.27:0/2503875327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3598119c30 con 0x7f35981046d0 2024-04-16T05:45:30.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.496+0000 7f359da00700 1 -- 172.21.15.27:0/2503875327 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359804c710 con 0x7f35981046d0 2024-04-16T05:45:30.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.497+0000 7f358ea00700 1 -- 172.21.15.27:0/2503875327 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 36) v1 ==== 92497+0+0 (secure 0 0 0) 0x7f358400aae0 con 0x7f35981046d0 2024-04-16T05:45:30.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.498+0000 7f358ea00700 1 --2- 172.21.15.27:0/2503875327 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f357c06e910 0x7f357c070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:30.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.498+0000 7f358ea00700 1 -- 172.21.15.27:0/2503875327 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f358408e420 con 0x7f35981046d0 2024-04-16T05:45:30.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.498+0000 7f3596a00700 1 --2- 172.21.15.27:0/2503875327 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f357c06e910 0x7f357c070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:45:30.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.499+0000 7f3596a00700 1 --2- 172.21.15.27:0/2503875327 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f357c06e910 0x7f357c070dc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f358800b3b0 tx=0x7f3588009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:45:30.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.503+0000 7f358ea00700 1 -- 172.21.15.27:0/2503875327 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3584057530 con 0x7f35981046d0 2024-04-16T05:45:30.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.737+0000 7f359da00700 1 -- 172.21.15.27:0/2503875327 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3598108a60 con 0x7f357c06e910 2024-04-16T05:45:30.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.749+0000 7f358ea00700 1 -- 172.21.15.27:0/2503875327 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f3598108a60 con 0x7f357c06e910 2024-04-16T05:45:30.748 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:45:30.748 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (110s) 24s ago 12m 13.4M - 0.25.0 c8568f914cd2 4ed63a8a6b33 2024-04-16T05:45:30.748 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (12m) 24s ago 12m 6665k - 17.2.7-847-g45317f95 b333581ccdf5 f1acb6959c63 2024-04-16T05:45:30.748 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (11m) 27s ago 11m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 e81efc20e8c0 2024-04-16T05:45:30.748 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (103s) 24s ago 12m 75.1M - 9.4.7 954c08fa6188 f7ee2a1afd12 2024-04-16T05:45:30.748 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (5m) 24s ago 5m 23.2M - 17.2.7-847-g45317f95 b333581ccdf5 0824e2621be5 2024-04-16T05:45:30.748 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (5m) 24s ago 5m 12.5M - 17.2.7-847-g45317f95 b333581ccdf5 b332a72cc594 2024-04-16T05:45:30.748 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (5m) 27s ago 5m 12.7M - 17.2.7-847-g45317f95 b333581ccdf5 f5e36815ddfa 2024-04-16T05:45:30.749 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (5m) 27s ago 5m 12.7M - 17.2.7-847-g45317f95 b333581ccdf5 755a73c65a7f 2024-04-16T05:45:30.749 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:8443,9283,8765 running (83s) 24s ago 14m 493M - 19.0.0-2884-g218a6914 a4e4811815e4 b69a0b08e12b 2024-04-16T05:45:30.749 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283,8765 running (32s) 27s ago 11m 68.2M - 19.0.0-2884-g218a6914 a4e4811815e4 32e27bcd7e42 2024-04-16T05:45:30.749 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (14m) 24s ago 14m 52.2M 2048M 17.2.7-847-g45317f95 b333581ccdf5 88575dbb8d98 2024-04-16T05:45:30.749 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (11m) 27s ago 11m 39.7M 2048M 17.2.7-847-g45317f95 b333581ccdf5 5df19eab6d21 2024-04-16T05:45:30.749 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (118s) 24s ago 12m 7562k - 1.5.0 0da6a335fe13 5e29e29868b0 2024-04-16T05:45:30.749 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (92s) 27s ago 11m 7759k - 1.5.0 0da6a335fe13 a360922bc621 2024-04-16T05:45:30.749 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (10m) 24s ago 10m 47.1M 1557M 17.2.7-847-g45317f95 b333581ccdf5 15e049573bca 2024-04-16T05:45:30.749 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (9m) 24s ago 9m 47.1M 1557M 17.2.7-847-g45317f95 b333581ccdf5 e146295f6500 2024-04-16T05:45:30.749 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (8m) 24s ago 8m 48.3M 1557M 17.2.7-847-g45317f95 b333581ccdf5 6212f6a18fa4 2024-04-16T05:45:30.749 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (8m) 27s ago 8m 45.2M 2581M 17.2.7-847-g45317f95 b333581ccdf5 d6724ad05f9f 2024-04-16T05:45:30.749 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (7m) 27s ago 7m 44.1M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7a59908a3e10 2024-04-16T05:45:30.749 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (7m) 27s ago 7m 43.4M 2581M 17.2.7-847-g45317f95 b333581ccdf5 8e4b60163907 2024-04-16T05:45:30.749 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (40s) 24s ago 11m 40.8M - 2.43.0 a07b618ecd1d e70f55ba5ec0 2024-04-16T05:45:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.752+0000 7f359da00700 1 -- 172.21.15.27:0/2503875327 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f357c06e910 msgr2=0x7f357c070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.752+0000 7f359da00700 1 --2- 172.21.15.27:0/2503875327 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f357c06e910 0x7f357c070dc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f358800b3b0 tx=0x7f3588009e30 comp rx=0 tx=0).stop 2024-04-16T05:45:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.752+0000 7f359da00700 1 -- 172.21.15.27:0/2503875327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35981046d0 msgr2=0x7f35981144a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.752+0000 7f359da00700 1 --2- 172.21.15.27:0/2503875327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35981046d0 0x7f35981144a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f358400c8c0 tx=0x7f35840044c0 comp rx=0 tx=0).stop 2024-04-16T05:45:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.752+0000 7f359da00700 1 -- 172.21.15.27:0/2503875327 shutdown_connections 2024-04-16T05:45:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.752+0000 7f359da00700 1 --2- 172.21.15.27:0/2503875327 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f357c06e910 0x7f357c070dc0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.752+0000 7f359da00700 1 --2- 172.21.15.27:0/2503875327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3598103420 0x7f3598113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.752+0000 7f359da00700 1 --2- 172.21.15.27:0/2503875327 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35981046d0 0x7f35981144a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.752+0000 7f359da00700 1 -- 172.21.15.27:0/2503875327 >> 172.21.15.27:0/2503875327 conn(0x7f3598072640 msgr2=0x7f3598073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:45:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.753+0000 7f359da00700 1 -- 172.21.15.27:0/2503875327 shutdown_connections 2024-04-16T05:45:30.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.753+0000 7f359da00700 1 -- 172.21.15.27:0/2503875327 wait complete. 2024-04-16T05:45:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.897+0000 7fef3ee00700 1 -- 172.21.15.27:0/3679584833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef380fffd0 msgr2=0x7fef381023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.897+0000 7fef3ee00700 1 --2- 172.21.15.27:0/3679584833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef380fffd0 0x7fef381023b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fef24004940 tx=0x7fef24009e30 comp rx=0 tx=0).stop 2024-04-16T05:45:30.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.897+0000 7fef3ee00700 1 -- 172.21.15.27:0/3679584833 shutdown_connections 2024-04-16T05:45:30.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.897+0000 7fef3ee00700 1 --2- 172.21.15.27:0/3679584833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef380fffd0 0x7fef381023b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:30.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.897+0000 7fef3ee00700 1 --2- 172.21.15.27:0/3679584833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef38066290 0x7fef380ffa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:30.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.897+0000 7fef3ee00700 1 -- 172.21.15.27:0/3679584833 >> 172.21.15.27:0/3679584833 conn(0x7fef380f74c0 msgr2=0x7fef380f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:45:30.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.897+0000 7fef3ee00700 1 -- 172.21.15.27:0/3679584833 shutdown_connections 2024-04-16T05:45:30.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.897+0000 7fef3ee00700 1 -- 172.21.15.27:0/3679584833 wait complete. 2024-04-16T05:45:30.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.898+0000 7fef3ee00700 1 Processor -- start 2024-04-16T05:45:30.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.898+0000 7fef3ee00700 1 -- start start 2024-04-16T05:45:30.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.899+0000 7fef3ee00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef38066290 0x7fef3810fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:30.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.899+0000 7fef3ee00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef380fffd0 0x7fef38110160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:30.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.899+0000 7fef3ee00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef38110760 con 0x7fef380fffd0 2024-04-16T05:45:30.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.899+0000 7fef3ee00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef38110fb0 con 0x7fef38066290 2024-04-16T05:45:30.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.899+0000 7fef37400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef380fffd0 0x7fef38110160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:45:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.899+0000 7fef37400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef380fffd0 0x7fef38110160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41322/0 (socket says 172.21.15.27:41322) 2024-04-16T05:45:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.899+0000 7fef37400700 1 -- 172.21.15.27:0/2037624179 learned_addr learned my addr 172.21.15.27:0/2037624179 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:45:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.899+0000 7fef37e00700 1 --2- 172.21.15.27:0/2037624179 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef38066290 0x7fef3810fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:45:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.899+0000 7fef37400700 1 -- 172.21.15.27:0/2037624179 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef38066290 msgr2=0x7fef3810fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.899+0000 7fef37400700 1 --2- 172.21.15.27:0/2037624179 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef38066290 0x7fef3810fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.899+0000 7fef37400700 1 -- 172.21.15.27:0/2037624179 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef24005040 con 0x7fef380fffd0 2024-04-16T05:45:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.900+0000 7fef37400700 1 --2- 172.21.15.27:0/2037624179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef380fffd0 0x7fef38110160 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fef2400b3b0 tx=0x7fef24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:45:30.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.905+0000 7fef2fe00700 1 -- 172.21.15.27:0/2037624179 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef2400e040 con 0x7fef380fffd0 2024-04-16T05:45:30.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.905+0000 7fef3ee00700 1 -- 172.21.15.27:0/2037624179 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef38111230 con 0x7fef380fffd0 2024-04-16T05:45:30.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.905+0000 7fef3ee00700 1 -- 172.21.15.27:0/2037624179 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef38111770 con 0x7fef380fffd0 2024-04-16T05:45:30.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.906+0000 7fef2fe00700 1 -- 172.21.15.27:0/2037624179 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef24004290 con 0x7fef380fffd0 2024-04-16T05:45:30.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.906+0000 7fef2fe00700 1 -- 172.21.15.27:0/2037624179 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef24013e80 con 0x7fef380fffd0 2024-04-16T05:45:30.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.907+0000 7fef3ee00700 1 -- 172.21.15.27:0/2037624179 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef180052e0 con 0x7fef380fffd0 2024-04-16T05:45:30.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.908+0000 7fef2fe00700 1 -- 172.21.15.27:0/2037624179 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 37) v1 ==== 92791+0+0 (secure 0 0 0) 0x7fef24013420 con 0x7fef380fffd0 2024-04-16T05:45:30.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.909+0000 7fef2fe00700 1 --2- 172.21.15.27:0/2037624179 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fef1c06eca0 0x7fef1c071150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:30.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.909+0000 7fef2fe00700 1 -- 172.21.15.27:0/2037624179 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fef2408fd40 con 0x7fef380fffd0 2024-04-16T05:45:30.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.909+0000 7fef37e00700 1 --2- 172.21.15.27:0/2037624179 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fef1c06eca0 0x7fef1c071150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:45:30.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.910+0000 7fef37e00700 1 --2- 172.21.15.27:0/2037624179 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fef1c06eca0 0x7fef1c071150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fef28007fd0 tx=0x7fef28009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:45:30.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:30.915+0000 7fef2fe00700 1 -- 172.21.15.27:0/2037624179 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fef2405c540 con 0x7fef380fffd0 2024-04-16T05:45:31.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:30 smithi027 ceph-mon[93635]: [16/Apr/2024:05:45:29] ENGINE Bus STARTING 2024-04-16T05:45:31.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:30 smithi027 ceph-mon[93635]: mgrmap e36: smithi027.qgzypi(active, since 1.02403s), standbys: smithi122.sgokkq 2024-04-16T05:45:31.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:30 smithi027 ceph-mon[93635]: from='client.34104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:45:31.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:30 smithi027 ceph-mon[93635]: pgmap v3: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:31.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:30 smithi027 ceph-mon[93635]: [16/Apr/2024:05:45:29] ENGINE Serving on https://172.21.15.27:7150 2024-04-16T05:45:31.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:30 smithi027 ceph-mon[93635]: [16/Apr/2024:05:45:30] ENGINE Serving on http://172.21.15.27:8765 2024-04-16T05:45:31.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:30 smithi027 ceph-mon[93635]: [16/Apr/2024:05:45:30] ENGINE Bus STARTED 2024-04-16T05:45:31.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:30 smithi027 ceph-mon[93635]: from='client.34128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:45:31.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.242+0000 7fef3ee00700 1 -- 172.21.15.27:0/2037624179 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fef18005860 con 0x7fef380fffd0 2024-04-16T05:45:31.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.242+0000 7fef2fe00700 1 -- 172.21.15.27:0/2037624179 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+843 (secure 0 0 0) 0x7fef2405c0d0 con 0x7fef380fffd0 2024-04-16T05:45:31.240 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:45:31.240 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:45:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 1, 2024-04-16T05:45:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 1 2024-04-16T05:45:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:45:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:45:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:45:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:45:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:45:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T05:45:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:45:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:45:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T05:45:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:45:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:45:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 11, 2024-04-16T05:45:31.241 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 3 2024-04-16T05:45:31.242 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:45:31.242 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:45:31.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.246+0000 7fef3ee00700 1 -- 172.21.15.27:0/2037624179 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fef1c06eca0 msgr2=0x7fef1c071150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:31.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.246+0000 7fef3ee00700 1 --2- 172.21.15.27:0/2037624179 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fef1c06eca0 0x7fef1c071150 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fef28007fd0 tx=0x7fef28009040 comp rx=0 tx=0).stop 2024-04-16T05:45:31.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.246+0000 7fef3ee00700 1 -- 172.21.15.27:0/2037624179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef380fffd0 msgr2=0x7fef38110160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:31.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.246+0000 7fef3ee00700 1 --2- 172.21.15.27:0/2037624179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef380fffd0 0x7fef38110160 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fef2400b3b0 tx=0x7fef24009e30 comp rx=0 tx=0).stop 2024-04-16T05:45:31.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.246+0000 7fef3ee00700 1 -- 172.21.15.27:0/2037624179 shutdown_connections 2024-04-16T05:45:31.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.246+0000 7fef3ee00700 1 --2- 172.21.15.27:0/2037624179 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fef1c06eca0 0x7fef1c071150 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:31.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.246+0000 7fef3ee00700 1 --2- 172.21.15.27:0/2037624179 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fef38066290 0x7fef3810fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:31.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.246+0000 7fef3ee00700 1 --2- 172.21.15.27:0/2037624179 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fef380fffd0 0x7fef38110160 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:31.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.246+0000 7fef3ee00700 1 -- 172.21.15.27:0/2037624179 >> 172.21.15.27:0/2037624179 conn(0x7fef380f74c0 msgr2=0x7fef380fbb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:45:31.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.246+0000 7fef3ee00700 1 -- 172.21.15.27:0/2037624179 shutdown_connections 2024-04-16T05:45:31.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.246+0000 7fef3ee00700 1 -- 172.21.15.27:0/2037624179 wait complete. 2024-04-16T05:45:31.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:30 smithi122 ceph-mon[35998]: [16/Apr/2024:05:45:29] ENGINE Bus STARTING 2024-04-16T05:45:31.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:30 smithi122 ceph-mon[35998]: mgrmap e36: smithi027.qgzypi(active, since 1.02403s), standbys: smithi122.sgokkq 2024-04-16T05:45:31.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:30 smithi122 ceph-mon[35998]: from='client.34104 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:45:31.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:30 smithi122 ceph-mon[35998]: pgmap v3: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:31.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:30 smithi122 ceph-mon[35998]: [16/Apr/2024:05:45:29] ENGINE Serving on https://172.21.15.27:7150 2024-04-16T05:45:31.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:30 smithi122 ceph-mon[35998]: [16/Apr/2024:05:45:30] ENGINE Serving on http://172.21.15.27:8765 2024-04-16T05:45:31.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:30 smithi122 ceph-mon[35998]: [16/Apr/2024:05:45:30] ENGINE Bus STARTED 2024-04-16T05:45:31.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:30 smithi122 ceph-mon[35998]: from='client.34128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:45:31.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.416+0000 7f167d800700 1 -- 172.21.15.27:0/2995970895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1678070890 msgr2=0x7f1678070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:31.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.416+0000 7f167d800700 1 --2- 172.21.15.27:0/2995970895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1678070890 0x7f1678070d00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1664004940 tx=0x7f1664009e30 comp rx=0 tx=0).stop 2024-04-16T05:45:31.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.416+0000 7f167d800700 1 -- 172.21.15.27:0/2995970895 shutdown_connections 2024-04-16T05:45:31.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.416+0000 7f167d800700 1 --2- 172.21.15.27:0/2995970895 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1678070890 0x7f1678070d00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:31.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.416+0000 7f167d800700 1 --2- 172.21.15.27:0/2995970895 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1678071e40 0x7f1678070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:31.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.416+0000 7f167d800700 1 -- 172.21.15.27:0/2995970895 >> 172.21.15.27:0/2995970895 conn(0x7f16780f8590 msgr2=0x7f16780fa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:45:31.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.416+0000 7f167d800700 1 -- 172.21.15.27:0/2995970895 shutdown_connections 2024-04-16T05:45:31.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.416+0000 7f167d800700 1 -- 172.21.15.27:0/2995970895 wait complete. 2024-04-16T05:45:31.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.417+0000 7f167d800700 1 Processor -- start 2024-04-16T05:45:31.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.417+0000 7f167d800700 1 -- start start 2024-04-16T05:45:31.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.418+0000 7f167d800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1678070890 0x7f167810fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:31.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.418+0000 7f167d800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1678071e40 0x7f1678110190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:31.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.418+0000 7f167d800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1678110790 con 0x7f1678070890 2024-04-16T05:45:31.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.418+0000 7f167d800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1678110fe0 con 0x7f1678071e40 2024-04-16T05:45:31.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.418+0000 7f1676a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1678070890 0x7f167810fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:45:31.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.418+0000 7f1676a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1678070890 0x7f167810fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41356/0 (socket says 172.21.15.27:41356) 2024-04-16T05:45:31.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.418+0000 7f1676a00700 1 -- 172.21.15.27:0/449947994 learned_addr learned my addr 172.21.15.27:0/449947994 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:45:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.418+0000 7f1676000700 1 --2- 172.21.15.27:0/449947994 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1678071e40 0x7f1678110190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:45:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.419+0000 7f1676a00700 1 -- 172.21.15.27:0/449947994 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1678071e40 msgr2=0x7f1678110190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.419+0000 7f1676a00700 1 --2- 172.21.15.27:0/449947994 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1678071e40 0x7f1678110190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.419+0000 7f1676a00700 1 -- 172.21.15.27:0/449947994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1664005040 con 0x7f1678070890 2024-04-16T05:45:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.419+0000 7f1676a00700 1 --2- 172.21.15.27:0/449947994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1678070890 0x7f167810fc50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f166800c8e0 tx=0x7f16680044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:45:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.419+0000 7f166ea00700 1 -- 172.21.15.27:0/449947994 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f166800a760 con 0x7f1678070890 2024-04-16T05:45:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.420+0000 7f166ea00700 1 -- 172.21.15.27:0/449947994 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f166800ad90 con 0x7f1678070890 2024-04-16T05:45:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.420+0000 7f166ea00700 1 -- 172.21.15.27:0/449947994 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16680113a0 con 0x7f1678070890 2024-04-16T05:45:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.420+0000 7f167d800700 1 -- 172.21.15.27:0/449947994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16781112c0 con 0x7f1678070890 2024-04-16T05:45:31.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.420+0000 7f167d800700 1 -- 172.21.15.27:0/449947994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1678111800 con 0x7f1678070890 2024-04-16T05:45:31.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.421+0000 7f167d800700 1 -- 172.21.15.27:0/449947994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f167805f5f0 con 0x7f1678070890 2024-04-16T05:45:31.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.422+0000 7f166ea00700 1 -- 172.21.15.27:0/449947994 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 37) v1 ==== 92791+0+0 (secure 0 0 0) 0x7f1668012400 con 0x7f1678070890 2024-04-16T05:45:31.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.423+0000 7f166ea00700 1 --2- 172.21.15.27:0/449947994 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f165c06eca0 0x7f165c071150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:31.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.423+0000 7f166ea00700 1 -- 172.21.15.27:0/449947994 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f166808e8f0 con 0x7f1678070890 2024-04-16T05:45:31.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.424+0000 7f1676000700 1 --2- 172.21.15.27:0/449947994 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f165c06eca0 0x7f165c071150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:45:31.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.424+0000 7f1676000700 1 --2- 172.21.15.27:0/449947994 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f165c06eca0 0x7f165c071150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1664009fd0 tx=0x7f1664009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:45:31.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.432+0000 7f166ea00700 1 -- 172.21.15.27:0/449947994 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f166805b200 con 0x7f1678070890 2024-04-16T05:45:31.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.722+0000 7f167d800700 1 -- 172.21.15.27:0/449947994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1678111450 con 0x7f1678070890 2024-04-16T05:45:31.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.722+0000 7f166ea00700 1 -- 172.21.15.27:0/449947994 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1880 (secure 0 0 0) 0x7f1678111450 con 0x7f1678070890 2024-04-16T05:45:31.720 INFO:teuthology.orchestra.run.smithi027.stdout:e16 2024-04-16T05:45:31.721 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:45:31.721 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:45:31.721 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:45:31.721 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:45:31.721 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:45:31.721 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:45:31.721 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 15 2024-04-16T05:45:31.721 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:45:31.721 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:45:31.721 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:40:32.701649+0000 2024-04-16T05:45:31.721 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:45:31.721 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:45:31.721 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:45:31.721 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:45:31.722 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:45:31.722 INFO:teuthology.orchestra.run.smithi027.stdout:max_xattr_size 65536 2024-04-16T05:45:31.722 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:45:31.722 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:45:31.722 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 46 2024-04-16T05:45:31.722 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:45:31.722 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:45:31.722 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:45:31.722 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=14462} 2024-04-16T05:45:31.722 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:45:31.722 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:45:31.722 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:45:31.722 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:45:31.722 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:45:31.723 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:45:31.723 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:45:31.723 INFO:teuthology.orchestra.run.smithi027.stdout:bal_rank_mask -1 2024-04-16T05:45:31.723 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:45:31.723 INFO:teuthology.orchestra.run.smithi027.stdout:qdb_cluster leader: 0 members: 2024-04-16T05:45:31.723 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:45:31.723 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:45:31.723 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:45:31.723 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:45:31.723 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:45:31.723 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{-1:14474} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2065690722,v1:172.21.15.27:6827/2065690722] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:45:31.723 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:24263} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:45:31.723 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:24273} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:45:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.725+0000 7f167d800700 1 -- 172.21.15.27:0/449947994 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f165c06eca0 msgr2=0x7f165c071150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.725+0000 7f167d800700 1 --2- 172.21.15.27:0/449947994 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f165c06eca0 0x7f165c071150 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1664009fd0 tx=0x7f1664009e30 comp rx=0 tx=0).stop 2024-04-16T05:45:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.725+0000 7f167d800700 1 -- 172.21.15.27:0/449947994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1678070890 msgr2=0x7f167810fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.725+0000 7f167d800700 1 --2- 172.21.15.27:0/449947994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1678070890 0x7f167810fc50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f166800c8e0 tx=0x7f16680044c0 comp rx=0 tx=0).stop 2024-04-16T05:45:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.725+0000 7f167d800700 1 -- 172.21.15.27:0/449947994 shutdown_connections 2024-04-16T05:45:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.725+0000 7f167d800700 1 --2- 172.21.15.27:0/449947994 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f165c06eca0 0x7f165c071150 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.725+0000 7f167d800700 1 --2- 172.21.15.27:0/449947994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1678070890 0x7f167810fc50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.725+0000 7f167d800700 1 --2- 172.21.15.27:0/449947994 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1678071e40 0x7f1678110190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.725+0000 7f167d800700 1 -- 172.21.15.27:0/449947994 >> 172.21.15.27:0/449947994 conn(0x7f16780f8590 msgr2=0x7f16780f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:45:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.725+0000 7f167d800700 1 -- 172.21.15.27:0/449947994 shutdown_connections 2024-04-16T05:45:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.725+0000 7f167d800700 1 -- 172.21.15.27:0/449947994 wait complete. 2024-04-16T05:45:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2024-04-16T05:45:31.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.871+0000 7fd823000700 1 -- 172.21.15.27:0/4114888696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd81c0662f0 msgr2=0x7fd81c0ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:31.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.871+0000 7fd823000700 1 --2- 172.21.15.27:0/4114888696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd81c0662f0 0x7fd81c0ffa30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd80c004940 tx=0x7fd80c009e30 comp rx=0 tx=0).stop 2024-04-16T05:45:31.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.871+0000 7fd823000700 1 -- 172.21.15.27:0/4114888696 shutdown_connections 2024-04-16T05:45:31.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.871+0000 7fd823000700 1 --2- 172.21.15.27:0/4114888696 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd81c0fff70 0x7fd81c102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:31.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.871+0000 7fd823000700 1 --2- 172.21.15.27:0/4114888696 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd81c0662f0 0x7fd81c0ffa30 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:31.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.871+0000 7fd823000700 1 -- 172.21.15.27:0/4114888696 >> 172.21.15.27:0/4114888696 conn(0x7fd81c0f74c0 msgr2=0x7fd81c0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:45:31.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.871+0000 7fd823000700 1 -- 172.21.15.27:0/4114888696 shutdown_connections 2024-04-16T05:45:31.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.872+0000 7fd823000700 1 -- 172.21.15.27:0/4114888696 wait complete. 2024-04-16T05:45:31.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.872+0000 7fd823000700 1 Processor -- start 2024-04-16T05:45:31.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.873+0000 7fd823000700 1 -- start start 2024-04-16T05:45:31.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.873+0000 7fd823000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd81c0662f0 0x7fd81c10fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:31.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.873+0000 7fd823000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd81c0fff70 0x7fd81c110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:31.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.873+0000 7fd823000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c110f60 con 0x7fd81c0fff70 2024-04-16T05:45:31.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.873+0000 7fd823000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81c1110d0 con 0x7fd81c0662f0 2024-04-16T05:45:31.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.873+0000 7fd81b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd81c0fff70 0x7fd81c110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:45:31.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.873+0000 7fd81b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd81c0fff70 0x7fd81c110250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41374/0 (socket says 172.21.15.27:41374) 2024-04-16T05:45:31.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.873+0000 7fd81b400700 1 -- 172.21.15.27:0/648985925 learned_addr learned my addr 172.21.15.27:0/648985925 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:45:31.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.874+0000 7fd81be00700 1 --2- 172.21.15.27:0/648985925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd81c0662f0 0x7fd81c10fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:45:31.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.874+0000 7fd81b400700 1 -- 172.21.15.27:0/648985925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd81c0662f0 msgr2=0x7fd81c10fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:31.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.874+0000 7fd81b400700 1 --2- 172.21.15.27:0/648985925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd81c0662f0 0x7fd81c10fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:31.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.874+0000 7fd81b400700 1 -- 172.21.15.27:0/648985925 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd80c005040 con 0x7fd81c0fff70 2024-04-16T05:45:31.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.874+0000 7fd81b400700 1 --2- 172.21.15.27:0/648985925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd81c0fff70 0x7fd81c110250 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd80800b790 tx=0x7fd808009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:45:31.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.874+0000 7fd813e00700 1 -- 172.21.15.27:0/648985925 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd80800f770 con 0x7fd81c0fff70 2024-04-16T05:45:31.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.875+0000 7fd813e00700 1 -- 172.21.15.27:0/648985925 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd80800fda0 con 0x7fd81c0fff70 2024-04-16T05:45:31.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.875+0000 7fd823000700 1 -- 172.21.15.27:0/648985925 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd81c1113b0 con 0x7fd81c0fff70 2024-04-16T05:45:31.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.875+0000 7fd813e00700 1 -- 172.21.15.27:0/648985925 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd80800d310 con 0x7fd81c0fff70 2024-04-16T05:45:31.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.875+0000 7fd823000700 1 -- 172.21.15.27:0/648985925 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd81c111970 con 0x7fd81c0fff70 2024-04-16T05:45:31.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.876+0000 7fd823000700 1 -- 172.21.15.27:0/648985925 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd81c05f5f0 con 0x7fd81c0fff70 2024-04-16T05:45:31.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.878+0000 7fd813e00700 1 -- 172.21.15.27:0/648985925 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 37) v1 ==== 92791+0+0 (secure 0 0 0) 0x7fd808010420 con 0x7fd81c0fff70 2024-04-16T05:45:31.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.879+0000 7fd813e00700 1 --2- 172.21.15.27:0/648985925 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd80006eca0 0x7fd800071150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:31.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.879+0000 7fd813e00700 1 -- 172.21.15.27:0/648985925 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd80808e8f0 con 0x7fd81c0fff70 2024-04-16T05:45:31.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.880+0000 7fd81be00700 1 --2- 172.21.15.27:0/648985925 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd80006eca0 0x7fd800071150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:45:31.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.881+0000 7fd81be00700 1 --2- 172.21.15.27:0/648985925 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd80006eca0 0x7fd800071150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd80c009fd0 tx=0x7fd80c009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:45:31.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:31.885+0000 7fd813e00700 1 -- 172.21.15.27:0/648985925 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd80805b200 con 0x7fd81c0fff70 2024-04-16T05:45:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:31 smithi027 ceph-mon[93635]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:45:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:31 smithi027 ceph-mon[93635]: pgmap v4: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:31 smithi027 ceph-mon[93635]: mgrmap e37: smithi027.qgzypi(active, since 2s), standbys: smithi122.sgokkq 2024-04-16T05:45:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:31 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/2037624179' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:45:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:31 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/449947994' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:45:32.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.140+0000 7fd823000700 1 -- 172.21.15.27:0/648985925 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd81c105930 con 0x7fd80006eca0 2024-04-16T05:45:32.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.141+0000 7fd813e00700 1 -- 172.21.15.27:0/648985925 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+337 (secure 0 0 0) 0x7fd81c105930 con 0x7fd80006eca0 2024-04-16T05:45:32.140 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:45:32.140 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T05:45:32.140 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:45:32.140 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:45:32.140 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [ 2024-04-16T05:45:32.140 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr" 2024-04-16T05:45:32.140 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-16T05:45:32.140 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "2/21 daemons upgraded", 2024-04-16T05:45:32.140 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "", 2024-04-16T05:45:32.140 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:45:32.140 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:45:32.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.144+0000 7fd823000700 1 -- 172.21.15.27:0/648985925 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd80006eca0 msgr2=0x7fd800071150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:32.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.144+0000 7fd823000700 1 --2- 172.21.15.27:0/648985925 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd80006eca0 0x7fd800071150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd80c009fd0 tx=0x7fd80c009e30 comp rx=0 tx=0).stop 2024-04-16T05:45:32.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.144+0000 7fd823000700 1 -- 172.21.15.27:0/648985925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd81c0fff70 msgr2=0x7fd81c110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:32.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.144+0000 7fd823000700 1 --2- 172.21.15.27:0/648985925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd81c0fff70 0x7fd81c110250 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd80800b790 tx=0x7fd808009250 comp rx=0 tx=0).stop 2024-04-16T05:45:32.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.144+0000 7fd823000700 1 -- 172.21.15.27:0/648985925 shutdown_connections 2024-04-16T05:45:32.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.144+0000 7fd823000700 1 --2- 172.21.15.27:0/648985925 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd80006eca0 0x7fd800071150 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:32.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.144+0000 7fd823000700 1 --2- 172.21.15.27:0/648985925 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd81c0662f0 0x7fd81c10fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:32.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.144+0000 7fd823000700 1 --2- 172.21.15.27:0/648985925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd81c0fff70 0x7fd81c110250 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:32.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.144+0000 7fd823000700 1 -- 172.21.15.27:0/648985925 >> 172.21.15.27:0/648985925 conn(0x7fd81c0f74c0 msgr2=0x7fd81c0fbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:45:32.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.144+0000 7fd823000700 1 -- 172.21.15.27:0/648985925 shutdown_connections 2024-04-16T05:45:32.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.144+0000 7fd823000700 1 -- 172.21.15.27:0/648985925 wait complete. 2024-04-16T05:45:32.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:31 smithi122 ceph-mon[35998]: from='client.34132 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:45:32.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:31 smithi122 ceph-mon[35998]: pgmap v4: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:32.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:31 smithi122 ceph-mon[35998]: mgrmap e37: smithi027.qgzypi(active, since 2s), standbys: smithi122.sgokkq 2024-04-16T05:45:32.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:31 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/2037624179' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:45:32.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:31 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/449947994' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:45:32.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.293+0000 7f4ff5800700 1 -- 172.21.15.27:0/2557406883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ff00fcef0 msgr2=0x7f4ff00fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:32.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.293+0000 7f4ff5800700 1 --2- 172.21.15.27:0/2557406883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ff00fcef0 0x7f4ff00fd320 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0004940 tx=0x7f4fe0009e30 comp rx=0 tx=0).stop 2024-04-16T05:45:32.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.294+0000 7f4ff5800700 1 -- 172.21.15.27:0/2557406883 shutdown_connections 2024-04-16T05:45:32.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.294+0000 7f4ff5800700 1 --2- 172.21.15.27:0/2557406883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ff00fe0e0 0x7f4ff00fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:32.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.294+0000 7f4ff5800700 1 --2- 172.21.15.27:0/2557406883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ff00fcef0 0x7f4ff00fd320 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:32.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.294+0000 7f4ff5800700 1 -- 172.21.15.27:0/2557406883 >> 172.21.15.27:0/2557406883 conn(0x7f4ff00f8530 msgr2=0x7f4ff00fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:45:32.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.294+0000 7f4ff5800700 1 -- 172.21.15.27:0/2557406883 shutdown_connections 2024-04-16T05:45:32.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.294+0000 7f4ff5800700 1 -- 172.21.15.27:0/2557406883 wait complete. 2024-04-16T05:45:32.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.295+0000 7f4ff5800700 1 Processor -- start 2024-04-16T05:45:32.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.295+0000 7f4ff5800700 1 -- start start 2024-04-16T05:45:32.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.295+0000 7f4ff5800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ff00fcef0 0x7f4ff0071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:32.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.295+0000 7f4ff5800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ff00fe0e0 0x7f4ff0070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:32.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.295+0000 7f4ff5800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff0070950 con 0x7f4ff00fcef0 2024-04-16T05:45:32.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.295+0000 7f4ff5800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ff0070ac0 con 0x7f4ff00fe0e0 2024-04-16T05:45:32.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.296+0000 7f4feea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ff00fcef0 0x7f4ff0071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:45:32.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.296+0000 7f4feea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ff00fcef0 0x7f4ff0071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41394/0 (socket says 172.21.15.27:41394) 2024-04-16T05:45:32.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.296+0000 7f4feea00700 1 -- 172.21.15.27:0/3748962664 learned_addr learned my addr 172.21.15.27:0/3748962664 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:45:32.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.296+0000 7f4fee000700 1 --2- 172.21.15.27:0/3748962664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ff00fe0e0 0x7f4ff0070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:45:32.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.296+0000 7f4feea00700 1 -- 172.21.15.27:0/3748962664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ff00fe0e0 msgr2=0x7f4ff0070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:32.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.296+0000 7f4feea00700 1 --2- 172.21.15.27:0/3748962664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ff00fe0e0 0x7f4ff0070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:32.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.296+0000 7f4feea00700 1 -- 172.21.15.27:0/3748962664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe0005040 con 0x7f4ff00fcef0 2024-04-16T05:45:32.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.296+0000 7f4fee000700 1 --2- 172.21.15.27:0/3748962664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ff00fe0e0 0x7f4ff0070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:45:32.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.296+0000 7f4feea00700 1 --2- 172.21.15.27:0/3748962664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ff00fcef0 0x7f4ff0071e40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4fe000b970 tx=0x7f4fe0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:45:32.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.297+0000 7f4fe6a00700 1 -- 172.21.15.27:0/3748962664 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fe000e040 con 0x7f4ff00fcef0 2024-04-16T05:45:32.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.297+0000 7f4fe6a00700 1 -- 172.21.15.27:0/3748962664 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4fe0004290 con 0x7f4ff00fcef0 2024-04-16T05:45:32.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.297+0000 7f4ff5800700 1 -- 172.21.15.27:0/3748962664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff0070d40 con 0x7f4ff00fcef0 2024-04-16T05:45:32.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.297+0000 7f4fe6a00700 1 -- 172.21.15.27:0/3748962664 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fe001c630 con 0x7f4ff00fcef0 2024-04-16T05:45:32.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.297+0000 7f4ff5800700 1 -- 172.21.15.27:0/3748962664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff0115a60 con 0x7f4ff00fcef0 2024-04-16T05:45:32.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.298+0000 7f4ff5800700 1 -- 172.21.15.27:0/3748962664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff005f5f0 con 0x7f4ff00fcef0 2024-04-16T05:45:32.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.299+0000 7f4fe6a00700 1 -- 172.21.15.27:0/3748962664 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 37) v1 ==== 92791+0+0 (secure 0 0 0) 0x7f4fe000bbf0 con 0x7f4ff00fcef0 2024-04-16T05:45:32.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.301+0000 7f4fe6a00700 1 --2- 172.21.15.27:0/3748962664 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4fd406eca0 0x7f4fd4071150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:45:32.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.301+0000 7f4fe6a00700 1 -- 172.21.15.27:0/3748962664 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f4fe008fd30 con 0x7f4ff00fcef0 2024-04-16T05:45:32.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.301+0000 7f4fee000700 1 --2- 172.21.15.27:0/3748962664 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4fd406eca0 0x7f4fd4071150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:45:32.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.302+0000 7f4fee000700 1 --2- 172.21.15.27:0/3748962664 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4fd406eca0 0x7f4fd4071150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc007b00 tx=0x7f4fdc008040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:45:32.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.306+0000 7f4fe6a00700 1 -- 172.21.15.27:0/3748962664 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4fe0058a00 con 0x7f4ff00fcef0 2024-04-16T05:45:32.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.631+0000 7f4ff5800700 1 -- 172.21.15.27:0/3748962664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f4ff0105a20 con 0x7f4ff00fcef0 2024-04-16T05:45:32.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.632+0000 7f4fe6a00700 1 -- 172.21.15.27:0/3748962664 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f4fe0013e60 con 0x7f4ff00fcef0 2024-04-16T05:45:32.631 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:45:32.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.635+0000 7f4ff5800700 1 -- 172.21.15.27:0/3748962664 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4fd406eca0 msgr2=0x7f4fd4071150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:32.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.635+0000 7f4ff5800700 1 --2- 172.21.15.27:0/3748962664 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4fd406eca0 0x7f4fd4071150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc007b00 tx=0x7f4fdc008040 comp rx=0 tx=0).stop 2024-04-16T05:45:32.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.635+0000 7f4ff5800700 1 -- 172.21.15.27:0/3748962664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ff00fcef0 msgr2=0x7f4ff0071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:45:32.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.635+0000 7f4ff5800700 1 --2- 172.21.15.27:0/3748962664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ff00fcef0 0x7f4ff0071e40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4fe000b970 tx=0x7f4fe0009e30 comp rx=0 tx=0).stop 2024-04-16T05:45:32.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.635+0000 7f4ff5800700 1 -- 172.21.15.27:0/3748962664 shutdown_connections 2024-04-16T05:45:32.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.635+0000 7f4ff5800700 1 --2- 172.21.15.27:0/3748962664 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4fd406eca0 0x7f4fd4071150 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:32.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.635+0000 7f4ff5800700 1 --2- 172.21.15.27:0/3748962664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4ff00fcef0 0x7f4ff0071e40 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:32.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.635+0000 7f4ff5800700 1 --2- 172.21.15.27:0/3748962664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ff00fe0e0 0x7f4ff0070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:45:32.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.635+0000 7f4ff5800700 1 -- 172.21.15.27:0/3748962664 >> 172.21.15.27:0/3748962664 conn(0x7f4ff00f8530 msgr2=0x7f4ff00f9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:45:32.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.635+0000 7f4ff5800700 1 -- 172.21.15.27:0/3748962664 shutdown_connections 2024-04-16T05:45:32.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:45:32.635+0000 7f4ff5800700 1 -- 172.21.15.27:0/3748962664 wait complete. 2024-04-16T05:45:33.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:32 smithi122 ceph-mon[35998]: from='client.34148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:45:33.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:32 smithi122 ceph-mon[35998]: from='client.? 172.21.15.27:0/3748962664' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:45:33.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:32 smithi027 ceph-mon[93635]: from='client.34148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:45:33.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:32 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/3748962664' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:45:34.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:33 smithi122 ceph-mon[35998]: pgmap v5: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:34.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:33 smithi122 ceph-mon[35998]: mgrmap e38: smithi027.qgzypi(active, since 4s), standbys: smithi122.sgokkq 2024-04-16T05:45:34.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:33 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:34.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:33 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:34.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:33 smithi027 ceph-mon[93635]: pgmap v5: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:34.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:33 smithi027 ceph-mon[93635]: mgrmap e38: smithi027.qgzypi(active, since 4s), standbys: smithi122.sgokkq 2024-04-16T05:45:34.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:33 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:34.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:33 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:36.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:35 smithi122 ceph-mon[35998]: pgmap v6: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:36.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:35 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:36.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:35 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:36.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:35 smithi027 ceph-mon[93635]: pgmap v6: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:36.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:35 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:36.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:35 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:38.230 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:37 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:38.230 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:37 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:38.231 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:37 smithi122 ceph-mon[35998]: pgmap v7: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-16T05:45:38.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:37 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:38.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:37 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:38.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:37 smithi027 ceph-mon[93635]: pgmap v7: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-16T05:45:39.639 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 180, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77310 2024-04-16T05:45:39.639 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 180, success, outsize: 120 2024-04-16T05:45:39.653 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 182, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77388 2024-04-16T05:45:39.653 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 182, success, outsize: 120 2024-04-16T05:45:39.673 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 184, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77469 2024-04-16T05:45:39.673 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 184, success, outsize: 120 2024-04-16T05:45:39.694 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 186, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77556 2024-04-16T05:45:39.694 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 186, success, outsize: 120 2024-04-16T05:45:39.716 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 188, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77640 2024-04-16T05:45:39.717 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 188, success, outsize: 120 2024-04-16T05:45:39.730 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 190, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77718 2024-04-16T05:45:39.730 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 190, success, outsize: 120 2024-04-16T05:45:39.744 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 192, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77799 2024-04-16T05:45:39.744 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 192, success, outsize: 120 2024-04-16T05:45:39.763 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 194, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77886 2024-04-16T05:45:39.763 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 194, success, outsize: 120 2024-04-16T05:45:39.781 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 196, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 77970 2024-04-16T05:45:39.781 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 196, success, outsize: 120 2024-04-16T05:45:39.795 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 198, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78048 2024-04-16T05:45:39.795 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 198, success, outsize: 120 2024-04-16T05:45:39.813 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 200, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78129 2024-04-16T05:45:39.813 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 200, success, outsize: 120 2024-04-16T05:45:39.832 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 202, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78216 2024-04-16T05:45:39.832 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 202, success, outsize: 120 2024-04-16T05:45:39.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 204, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78300 2024-04-16T05:45:39.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 204, success, outsize: 120 2024-04-16T05:45:39.870 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 206, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78378 2024-04-16T05:45:39.870 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 206, success, outsize: 120 2024-04-16T05:45:39.883 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 208, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78459 2024-04-16T05:45:39.884 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 208, success, outsize: 120 2024-04-16T05:45:39.903 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 210, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78546 2024-04-16T05:45:39.903 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 210, success, outsize: 120 2024-04-16T05:45:39.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 212, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78630 2024-04-16T05:45:39.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 212, success, outsize: 120 2024-04-16T05:45:39.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 214, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78708 2024-04-16T05:45:39.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 214, success, outsize: 120 2024-04-16T05:45:39.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 216, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78789 2024-04-16T05:45:39.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 216, success, outsize: 120 2024-04-16T05:45:39.958 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 218, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78876 2024-04-16T05:45:39.959 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 218, success, outsize: 120 2024-04-16T05:45:39.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 220, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 78960 2024-04-16T05:45:39.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 220, success, outsize: 120 2024-04-16T05:45:39.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 222, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79040 2024-04-16T05:45:39.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 222, success, outsize: 120 2024-04-16T05:45:39.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 224, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79121 2024-04-16T05:45:39.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 224, success, outsize: 120 2024-04-16T05:45:40.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 226, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79209 2024-04-16T05:45:40.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 226, success, outsize: 120 2024-04-16T05:45:40.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 228, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79293 2024-04-16T05:45:40.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 228, success, outsize: 120 2024-04-16T05:45:40.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 230, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79371 2024-04-16T05:45:40.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 230, success, outsize: 120 2024-04-16T05:45:40.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 232, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79452 2024-04-16T05:45:40.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 232, success, outsize: 120 2024-04-16T05:45:40.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 234, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 79539 2024-04-16T05:45:40.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 234, success, outsize: 120 2024-04-16T05:45:40.177 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:40 smithi122 ceph-mon[35998]: pgmap v8: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-16T05:45:40.177 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:40 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:40.177 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:40 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:40.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:40 smithi027 ceph-mon[93635]: pgmap v8: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-16T05:45:40.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:40.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:42.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:41 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:42.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:41 smithi122 ceph-mon[35998]: pgmap v9: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-16T05:45:42.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:41 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:42.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:41 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:45:42.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:41 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:45:42.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:41 smithi122 ceph-mon[35998]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-16T05:45:42.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:41 smithi122 ceph-mon[35998]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-16T05:45:42.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:41 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:42.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:41 smithi027 ceph-mon[93635]: pgmap v9: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-16T05:45:42.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:41 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:42.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:41 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:45:42.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:41 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:45:42.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:41 smithi027 ceph-mon[93635]: Updating smithi027:/etc/ceph/ceph.conf 2024-04-16T05:45:42.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:41 smithi027 ceph-mon[93635]: Updating smithi122:/etc/ceph/ceph.conf 2024-04-16T05:45:43.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:42 smithi027 ceph-mon[93635]: Updating smithi122:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:45:43.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:42 smithi027 ceph-mon[93635]: Updating smithi027:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:45:43.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:42 smithi027 ceph-mon[93635]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-16T05:45:43.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:42 smithi027 ceph-mon[93635]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-16T05:45:43.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:42 smithi122 ceph-mon[35998]: Updating smithi122:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:45:43.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:42 smithi122 ceph-mon[35998]: Updating smithi027:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.conf 2024-04-16T05:45:43.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:42 smithi122 ceph-mon[35998]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2024-04-16T05:45:43.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:42 smithi122 ceph-mon[35998]: Updating smithi027:/etc/ceph/ceph.client.admin.keyring 2024-04-16T05:45:44.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:43 smithi122 ceph-mon[35998]: pgmap v10: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T05:45:44.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:43 smithi122 ceph-mon[35998]: Updating smithi122:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T05:45:44.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:43 smithi122 ceph-mon[35998]: Updating smithi027:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T05:45:44.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:43 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:44.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:43 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:44.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:43 smithi027 ceph-mon[93635]: pgmap v10: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T05:45:44.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:43 smithi027 ceph-mon[93635]: Updating smithi122:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T05:45:44.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:43 smithi027 ceph-mon[93635]: Updating smithi027:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/config/ceph.client.admin.keyring 2024-04-16T05:45:44.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:43 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:44.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:43 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:45.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:44 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:45:45.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:44 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:45.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:44 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:45.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:44 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:45.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:44 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:45:45.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:44 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:45:45.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:44 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-16T05:45:45.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:45:45.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:45.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:45.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:45.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:45:45.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:45:45.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-16T05:45:46.210 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:46 smithi122 ceph-mon[35998]: pgmap v11: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T05:45:46.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:46 smithi027 ceph-mon[93635]: pgmap v11: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T05:45:47.963 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:47 smithi122 ceph-mon[35998]: Upgrade: Updating mon.smithi122 2024-04-16T05:45:47.964 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:47 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:47.964 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:47 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T05:45:47.964 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:47 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T05:45:47.964 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:47 smithi122 ceph-mon[35998]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:45:47.964 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:47 smithi122 ceph-mon[35998]: Deploying daemon mon.smithi122 on smithi122 2024-04-16T05:45:47.964 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:47 smithi122 ceph-mon[35998]: pgmap v12: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T05:45:48.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:47 smithi027 ceph-mon[93635]: Upgrade: Updating mon.smithi122 2024-04-16T05:45:48.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:47 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:48.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:47 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T05:45:48.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:47 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T05:45:48.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:47 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:45:48.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:47 smithi027 ceph-mon[93635]: Deploying daemon mon.smithi122 on smithi122 2024-04-16T05:45:48.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:47 smithi027 ceph-mon[93635]: pgmap v12: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-16T05:45:49.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:49 smithi122 systemd[1]: Stopping Ceph mon.smithi122 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T05:45:50.008 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:49 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi122[35992]: 2024-04-16T05:45:49.702+0000 7f37bb400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi122 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T05:45:50.008 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:49 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi122[35992]: 2024-04-16T05:45:49.702+0000 7f37bb400700 -1 mon.smithi122@1(peon) e2 *** Got Signal Terminated *** 2024-04-16T05:45:50.008 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:49 smithi122 podman[80770]: 2024-04-16 05:45:49.734864217 +0000 UTC m=+0.185061826 container died 5df19eab6d214c734d17d4f0a3c4f2c1cdf7e7b05541c0171eb1bdf91fd855bd (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi122, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-64502-quincy-3-45317f9, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-16T05:45:50.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:50 smithi122 podman[80770]: 2024-04-16 05:45:50.121503935 +0000 UTC m=+0.571701537 container cleanup 5df19eab6d214c734d17d4f0a3c4f2c1cdf7e7b05541c0171eb1bdf91fd855bd (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi122, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T05:45:50.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:50 smithi122 bash[80770]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi122 2024-04-16T05:45:50.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:50 smithi122 podman[80784]: 2024-04-16 05:45:50.354500338 +0000 UTC m=+0.608026619 container remove 5df19eab6d214c734d17d4f0a3c4f2c1cdf7e7b05541c0171eb1bdf91fd855bd (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi122, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-64502-quincy-3-45317f9, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-16T05:45:51.177 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:50 smithi122 systemd[1]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@mon.smithi122.service: Deactivated successfully. 2024-04-16T05:45:51.178 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:50 smithi122 systemd[1]: Stopped Ceph mon.smithi122 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968. 2024-04-16T05:45:51.178 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:50 smithi122 systemd[1]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@mon.smithi122.service: Consumed 14.271s CPU time. 2024-04-16T05:45:51.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 systemd[1]: Starting Ceph mon.smithi122 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T05:45:51.910 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 podman[80911]: 2024-04-16 05:45:51.496492066 +0000 UTC m=+0.034230651 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T05:45:51.910 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 podman[80911]: 2024-04-16 05:45:51.633113772 +0000 UTC m=+0.170852394 container create 5db5f3037ed0a5942a64aaa1dc8158b261b55fd6ed8e5d301e28c68e845aac41 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi122, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True) 2024-04-16T05:45:51.910 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 podman[80911]: 2024-04-16 05:45:51.911468376 +0000 UTC m=+0.449207005 container init 5db5f3037ed0a5942a64aaa1dc8158b261b55fd6ed8e5d301e28c68e845aac41 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi122, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0) 2024-04-16T05:45:52.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 podman[80911]: 2024-04-16 05:45:51.924200607 +0000 UTC m=+0.461939262 container start 5db5f3037ed0a5942a64aaa1dc8158b261b55fd6ed8e5d301e28c68e845aac41 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi122, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-16T05:45:52.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: set uid:gid to 167:167 (ceph:ceph) 2024-04-16T05:45:52.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev), process ceph-mon, pid 2 2024-04-16T05:45:52.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: pidfile_write: ignore empty --pid-file 2024-04-16T05:45:52.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: load: jerasure load: lrc 2024-04-16T05:45:52.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: RocksDB version: 7.9.2 2024-04-16T05:45:52.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Git sha 0 2024-04-16T05:45:52.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Compile date 2024-04-12 04:47:35 2024-04-16T05:45:52.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: DB SUMMARY 2024-04-16T05:45:52.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: DB Session ID: FGTJ1XVKDIJEMWSVHIVM 2024-04-16T05:45:52.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: CURRENT file: CURRENT 2024-04-16T05:45:52.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: IDENTITY file: IDENTITY 2024-04-16T05:45:52.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1072 Bytes 2024-04-16T05:45:52.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi122/store.db dir, Total Num: 1, files: 000027.sst 2024-04-16T05:45:52.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi122/store.db: 000025.log size: 6148790 ; 2024-04-16T05:45:52.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.error_if_exists: 0 2024-04-16T05:45:52.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.create_if_missing: 0 2024-04-16T05:45:52.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.paranoid_checks: 1 2024-04-16T05:45:52.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-16T05:45:52.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-16T05:45:52.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-16T05:45:52.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.env: 0x55a92b8f4cc0 2024-04-16T05:45:52.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.fs: PosixFileSystem 2024-04-16T05:45:52.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.info_log: 0x55a96644bae0 2024-04-16T05:45:52.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_file_opening_threads: 16 2024-04-16T05:45:52.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.statistics: (nil) 2024-04-16T05:45:52.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.use_fsync: 0 2024-04-16T05:45:52.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_log_file_size: 0 2024-04-16T05:45:52.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-16T05:45:52.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-16T05:45:52.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.keep_log_file_num: 1000 2024-04-16T05:45:52.275 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.recycle_log_file_num: 0 2024-04-16T05:45:52.275 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.allow_fallocate: 1 2024-04-16T05:45:52.275 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.allow_mmap_reads: 0 2024-04-16T05:45:52.275 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.allow_mmap_writes: 0 2024-04-16T05:45:52.275 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.use_direct_reads: 0 2024-04-16T05:45:52.275 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-16T05:45:52.275 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.create_missing_column_families: 0 2024-04-16T05:45:52.275 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.db_log_dir: 2024-04-16T05:45:52.275 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.wal_dir: 2024-04-16T05:45:52.275 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-16T05:45:52.275 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-16T05:45:52.276 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-16T05:45:52.276 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-16T05:45:52.276 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-16T05:45:52.276 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-16T05:45:52.276 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.advise_random_on_open: 1 2024-04-16T05:45:52.276 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.db_write_buffer_size: 0 2024-04-16T05:45:52.276 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.write_buffer_manager: 0x55a96641c1e0 2024-04-16T05:45:52.276 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-16T05:45:52.276 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-16T05:45:52.276 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-16T05:45:52.276 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.rate_limiter: (nil) 2024-04-16T05:45:52.276 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-16T05:45:52.276 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.wal_recovery_mode: 2 2024-04-16T05:45:52.276 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.enable_thread_tracking: 0 2024-04-16T05:45:52.276 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.enable_pipelined_write: 0 2024-04-16T05:45:52.277 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.unordered_write: 0 2024-04-16T05:45:52.277 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-16T05:45:52.277 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-16T05:45:52.277 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-16T05:45:52.277 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-16T05:45:52.277 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.row_cache: None 2024-04-16T05:45:52.277 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.wal_filter: None 2024-04-16T05:45:52.277 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-16T05:45:52.277 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.allow_ingest_behind: 0 2024-04-16T05:45:52.277 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.two_write_queues: 0 2024-04-16T05:45:52.277 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.manual_wal_flush: 0 2024-04-16T05:45:52.277 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.wal_compression: 0 2024-04-16T05:45:52.277 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.atomic_flush: 0 2024-04-16T05:45:52.277 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-16T05:45:52.277 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-16T05:45:52.278 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-16T05:45:52.278 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.log_readahead_size: 0 2024-04-16T05:45:52.278 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-16T05:45:52.278 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.best_efforts_recovery: 0 2024-04-16T05:45:52.279 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-16T05:45:52.279 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-16T05:45:52.279 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.allow_data_in_errors: 0 2024-04-16T05:45:52.279 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.db_host_id: __hostname__ 2024-04-16T05:45:52.279 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-16T05:45:52.279 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_background_jobs: 2 2024-04-16T05:45:52.279 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_background_compactions: -1 2024-04-16T05:45:52.279 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_subcompactions: 1 2024-04-16T05:45:52.279 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-16T05:45:52.279 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-16T05:45:52.279 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-16T05:45:52.279 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_total_wal_size: 0 2024-04-16T05:45:52.279 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-16T05:45:52.279 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-16T05:45:52.280 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-16T05:45:52.280 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-16T05:45:52.280 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_open_files: -1 2024-04-16T05:45:52.280 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.bytes_per_sync: 0 2024-04-16T05:45:52.280 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-16T05:45:52.280 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-16T05:45:52.280 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compaction_readahead_size: 0 2024-04-16T05:45:52.280 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_background_flushes: -1 2024-04-16T05:45:52.280 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Compression algorithms supported: 2024-04-16T05:45:52.280 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-16T05:45:52.280 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: kZSTD supported: 0 2024-04-16T05:45:52.280 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: kXpressCompression supported: 0 2024-04-16T05:45:52.280 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: kLZ4HCCompression supported: 1 2024-04-16T05:45:52.280 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: kZlibCompression supported: 1 2024-04-16T05:45:52.280 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: kSnappyCompression supported: 1 2024-04-16T05:45:52.281 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: kLZ4Compression supported: 1 2024-04-16T05:45:52.281 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: kBZip2Compression supported: 0 2024-04-16T05:45:52.281 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-16T05:45:52.281 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-16T05:45:52.281 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi122/store.db/MANIFEST-000009 2024-04-16T05:45:52.281 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-16T05:45:52.281 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-16T05:45:52.281 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.merge_operator: 2024-04-16T05:45:52.281 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compaction_filter: None 2024-04-16T05:45:52.281 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compaction_filter_factory: None 2024-04-16T05:45:52.281 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.sst_partitioner_factory: None 2024-04-16T05:45:52.281 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-16T05:45:52.281 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-16T05:45:52.281 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a96644bcc0) 2024-04-16T05:45:52.281 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: cache_index_and_filter_blocks: 1 2024-04-16T05:45:52.281 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-16T05:45:52.282 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-16T05:45:52.282 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: pin_top_level_index_and_filter: 1 2024-04-16T05:45:52.282 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: index_type: 0 2024-04-16T05:45:52.282 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: data_block_index_type: 0 2024-04-16T05:45:52.282 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: index_shortening: 1 2024-04-16T05:45:52.282 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-16T05:45:52.282 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: checksum: 4 2024-04-16T05:45:52.282 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: no_block_cache: 0 2024-04-16T05:45:52.282 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_cache: 0x55a96644f090 2024-04-16T05:45:52.282 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_cache_name: BinnedLRUCache 2024-04-16T05:45:52.282 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_cache_options: 2024-04-16T05:45:52.282 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: capacity : 536870912 2024-04-16T05:45:52.282 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: num_shard_bits : 4 2024-04-16T05:45:52.282 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: strict_capacity_limit : 0 2024-04-16T05:45:52.282 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: high_pri_pool_ratio: 0.000 2024-04-16T05:45:52.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_cache_compressed: (nil) 2024-04-16T05:45:52.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: persistent_cache: (nil) 2024-04-16T05:45:52.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_size: 4096 2024-04-16T05:45:52.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_size_deviation: 10 2024-04-16T05:45:52.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_restart_interval: 16 2024-04-16T05:45:52.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: index_block_restart_interval: 1 2024-04-16T05:45:52.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: metadata_block_size: 4096 2024-04-16T05:45:52.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: partition_filters: 0 2024-04-16T05:45:52.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: use_delta_encoding: 1 2024-04-16T05:45:52.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: filter_policy: bloomfilter 2024-04-16T05:45:52.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: whole_key_filtering: 1 2024-04-16T05:45:52.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: verify_compression: 0 2024-04-16T05:45:52.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: read_amp_bytes_per_bit: 0 2024-04-16T05:45:52.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: format_version: 5 2024-04-16T05:45:52.283 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: enable_index_compression: 1 2024-04-16T05:45:52.284 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: block_align: 0 2024-04-16T05:45:52.284 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: max_auto_readahead_size: 262144 2024-04-16T05:45:52.284 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: prepopulate_block_cache: 0 2024-04-16T05:45:52.284 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: initial_auto_readahead_size: 8192 2024-04-16T05:45:52.284 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: num_file_reads_for_auto_readahead: 2 2024-04-16T05:45:52.284 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.write_buffer_size: 33554432 2024-04-16T05:45:52.284 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_write_buffer_number: 2 2024-04-16T05:45:52.284 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compression: NoCompression 2024-04-16T05:45:52.284 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.bottommost_compression: Disabled 2024-04-16T05:45:52.284 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.prefix_extractor: nullptr 2024-04-16T05:45:52.284 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-16T05:45:52.284 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.num_levels: 7 2024-04-16T05:45:52.284 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-16T05:45:52.284 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-16T05:45:52.284 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-16T05:45:52.285 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-16T05:45:52.285 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-16T05:45:52.285 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-16T05:45:52.285 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-16T05:45:52.285 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-16T05:45:52.285 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-16T05:45:52.285 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-16T05:45:52.285 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-16T05:45:52.285 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-16T05:45:52.285 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-16T05:45:52.285 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compression_opts.level: 32767 2024-04-16T05:45:52.285 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compression_opts.strategy: 0 2024-04-16T05:45:52.285 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-16T05:45:52.285 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-16T05:45:52.285 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-16T05:45:52.286 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-16T05:45:52.286 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compression_opts.enabled: false 2024-04-16T05:45:52.286 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-16T05:45:52.286 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-16T05:45:52.286 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-16T05:45:52.286 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-16T05:45:52.286 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.target_file_size_base: 67108864 2024-04-16T05:45:52.286 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-16T05:45:52.286 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-16T05:45:52.286 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-16T05:45:52.286 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-16T05:45:52.286 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-16T05:45:52.286 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-16T05:45:52.286 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-16T05:45:52.286 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-16T05:45:52.286 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-16T05:45:52.287 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-16T05:45:52.287 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-16T05:45:52.287 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-16T05:45:52.287 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-16T05:45:52.287 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-16T05:45:52.288 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.arena_block_size: 1048576 2024-04-16T05:45:52.288 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-16T05:45:52.288 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-16T05:45:52.288 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.disable_auto_compactions: 0 2024-04-16T05:45:52.288 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-16T05:45:52.288 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-16T05:45:52.288 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-16T05:45:52.288 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-16T05:45:52.288 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-16T05:45:52.288 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-16T05:45:52.288 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-16T05:45:52.288 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-16T05:45:52.288 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-16T05:45:52.288 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-16T05:45:52.289 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-16T05:45:52.289 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.inplace_update_support: 0 2024-04-16T05:45:52.289 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-16T05:45:52.289 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-16T05:45:52.289 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-16T05:45:52.289 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-16T05:45:52.289 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.bloom_locality: 0 2024-04-16T05:45:52.289 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.max_successive_merges: 0 2024-04-16T05:45:52.289 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-16T05:45:52.289 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.paranoid_file_checks: 0 2024-04-16T05:45:52.289 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.force_consistency_checks: 1 2024-04-16T05:45:52.289 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.report_bg_io_stats: 0 2024-04-16T05:45:52.289 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.ttl: 2592000 2024-04-16T05:45:52.289 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-16T05:45:52.289 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-16T05:45:52.290 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-16T05:45:52.290 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.enable_blob_files: false 2024-04-16T05:45:52.290 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.min_blob_size: 0 2024-04-16T05:45:52.290 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.blob_file_size: 268435456 2024-04-16T05:45:52.290 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-16T05:45:52.290 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-16T05:45:52.290 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-16T05:45:52.290 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-16T05:45:52.290 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-16T05:45:52.290 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.blob_file_starting_level: 0 2024-04-16T05:45:52.290 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-16T05:45:52.290 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-04-16T05:45:52.290 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-16T05:45:52.290 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi122/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 11828, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-16T05:45:52.290 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-04-16T05:45:52.291 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ec05ef60-e035-4e5f-b4a8-0e053e1ad4c8 2024-04-16T05:45:52.291 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713246351980960, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-04-16T05:45:52.291 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:51 smithi122 ceph-mon[80930]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-04-16T05:45:52.291 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:52 smithi122 ceph-mon[80930]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713246352004019, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 3730732, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11833, "largest_seqno": 13386, "table_properties": {"data_size": 3723524, "index_size": 4499, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1669, "raw_key_size": 14940, "raw_average_key_size": 23, "raw_value_size": 3709605, "raw_average_value_size": 5760, "num_data_blocks": 205, "num_entries": 644, "num_filter_entries": 644, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713246351, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ec05ef60-e035-4e5f-b4a8-0e053e1ad4c8", "db_session_id": "FGTJ1XVKDIJEMWSVHIVM", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-04-16T05:45:52.291 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:52 smithi122 ceph-mon[80930]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713246352004358, "job": 1, "event": "recovery_finished"} 2024-04-16T05:45:52.291 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:52 smithi122 ceph-mon[80930]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-04-16T05:45:52.291 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:52 smithi122 ceph-mon[80930]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-16T05:45:52.291 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:52 smithi122 ceph-mon[80930]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi122/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-16T05:45:52.291 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:52 smithi122 ceph-mon[80930]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a96653e000 2024-04-16T05:45:52.291 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:52 smithi122 ceph-mon[80930]: rocksdb: DB pointer 0x55a966528000 2024-04-16T05:45:52.291 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:52 smithi122 ceph-mon[80930]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-16T05:45:52.291 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:52 smithi122 ceph-mon[80930]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-16T05:45:52.291 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ** DB Stats ** 2024-04-16T05:45:52.291 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T05:45:52.292 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-16T05:45:52.292 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T05:45:52.292 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T05:45:52.292 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-16T05:45:52.292 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-16T05:45:52.292 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-16T05:45:52.292 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-16T05:45:52.292 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ** Compaction Stats [default] ** 2024-04-16T05:45:52.292 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-16T05:45:52.292 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-16T05:45:52.292 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: L0 1/0 3.56 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 455.4 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-16T05:45:52.292 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: L6 1/0 5.48 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-16T05:45:52.292 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Sum 2/0 9.04 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 455.4 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-16T05:45:52.292 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 455.4 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-16T05:45:52.292 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-16T05:45:52.293 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ** Compaction Stats [default] ** 2024-04-16T05:45:52.293 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-16T05:45:52.293 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-16T05:45:52.293 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 455.4 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-04-16T05:45:52.293 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-16T05:45:52.293 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-16T05:45:52.293 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-16T05:45:52.293 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-16T05:45:52.293 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-04-16T05:45:52.293 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-16T05:45:52.293 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-16T05:45:52.293 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-16T05:45:52.293 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-16T05:45:52.293 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Cumulative compaction: 0.00 GB write, 121.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T05:45:52.293 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Interval compaction: 0.00 GB write, 121.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-16T05:45:52.293 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-16T05:45:52.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Block cache BinnedLRUCache@0x55a96644f090#2 capacity: 512.00 MB usage: 6.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.6e-05 secs_since: 0 2024-04-16T05:45:52.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,4.58 KB,0.000873208%) Misc(1,0.00 KB,0%) 2024-04-16T05:45:52.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-16T05:45:52.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-16T05:45:52.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:52 smithi122 ceph-mon[80930]: starting mon.smithi122 rank 1 at public addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] at bind addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi122 fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:45:52.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:52 smithi122 ceph-mon[80930]: mon.smithi122@-1(???) e2 preinit fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:45:52.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:52 smithi122 ceph-mon[80930]: mon.smithi122@-1(???).mds e16 new map 2024-04-16T05:45:52.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:52 smithi122 ceph-mon[80930]: mon.smithi122@-1(???).mds e16 print_map 2024-04-16T05:45:52.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: e16 2024-04-16T05:45:52.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:45:52.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:45:52.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: legacy client fscid: 1 2024-04-16T05:45:52.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-16T05:45:52.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Filesystem 'cephfs' (1) 2024-04-16T05:45:52.294 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: fs_name cephfs 2024-04-16T05:45:52.295 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: epoch 15 2024-04-16T05:45:52.295 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:45:52.295 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:45:52.295 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: modified 2024-04-16T05:40:32.701649+0000 2024-04-16T05:45:52.295 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: tableserver 0 2024-04-16T05:45:52.295 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: root 0 2024-04-16T05:45:52.295 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: session_timeout 60 2024-04-16T05:45:52.295 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: session_autoclose 300 2024-04-16T05:45:52.295 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: max_file_size 1099511627776 2024-04-16T05:45:52.295 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: max_xattr_size 65536 2024-04-16T05:45:52.295 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: required_client_features {} 2024-04-16T05:45:52.295 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: last_failure 0 2024-04-16T05:45:52.295 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: last_failure_osd_epoch 46 2024-04-16T05:45:52.295 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:45:52.295 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: max_mds 1 2024-04-16T05:45:52.296 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: in 0 2024-04-16T05:45:52.296 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: up {0=14462} 2024-04-16T05:45:52.296 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: failed 2024-04-16T05:45:52.296 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: damaged 2024-04-16T05:45:52.296 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: stopped 2024-04-16T05:45:52.296 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: data_pools [3] 2024-04-16T05:45:52.296 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: metadata_pool 2 2024-04-16T05:45:52.296 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: inline_data disabled 2024-04-16T05:45:52.296 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: balancer 2024-04-16T05:45:52.296 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: bal_rank_mask -1 2024-04-16T05:45:52.296 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: standby_count_wanted 1 2024-04-16T05:45:52.296 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: qdb_cluster leader: 0 members: 2024-04-16T05:45:52.296 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: [mds.cephfs.smithi027.ejoytk{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:45:52.296 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-16T05:45:52.296 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-16T05:45:52.297 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: Standby daemons: 2024-04-16T05:45:52.297 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: 2024-04-16T05:45:52.297 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: [mds.cephfs.smithi027.hkhpuq{-1:14474} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2065690722,v1:172.21.15.27:6827/2065690722] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:45:52.297 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: [mds.cephfs.smithi122.lojjdp{-1:24263} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:45:52.297 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout: [mds.cephfs.smithi122.rulewi{-1:24273} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:45:52.297 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:52 smithi122 ceph-mon[80930]: mon.smithi122@-1(???).osd e49 crush map has features 3314933000852226048, adjusting msgr requires 2024-04-16T05:45:52.297 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:52 smithi122 ceph-mon[80930]: mon.smithi122@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T05:45:52.297 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:52 smithi122 ceph-mon[80930]: mon.smithi122@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T05:45:52.297 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:52 smithi122 ceph-mon[80930]: mon.smithi122@-1(???).osd e49 crush map has features 288514051259236352, adjusting msgr requires 2024-04-16T05:45:52.297 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:52 smithi122 ceph-mon[80930]: mon.smithi122@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-04-16T05:45:52.297 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:52 smithi122 bash[80911]: 5db5f3037ed0a5942a64aaa1dc8158b261b55fd6ed8e5d301e28c68e845aac41 2024-04-16T05:45:52.297 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:52 smithi122 systemd[1]: Started Ceph mon.smithi122 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968. 2024-04-16T05:45:53.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:53 smithi122 ceph-mon[80930]: pgmap v13: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:53.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:53 smithi027 ceph-mon[93635]: pgmap v13: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:54.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:54 smithi122 ceph-mon[80930]: pgmap v15: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:54.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:54 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-16T05:45:54.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:54 smithi122 ceph-mon[80930]: mon.smithi027 calling monitor election 2024-04-16T05:45:54.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:54 smithi122 ceph-mon[80930]: mon.smithi122 calling monitor election 2024-04-16T05:45:54.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:54 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:45:54.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:54 smithi122 ceph-mon[80930]: mon.smithi027 is new leader, mons smithi027,smithi122 in quorum (ranks 0,1) 2024-04-16T05:45:54.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:54 smithi122 ceph-mon[80930]: monmap epoch 3 2024-04-16T05:45:54.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:54 smithi122 ceph-mon[80930]: fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:45:54.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:54 smithi122 ceph-mon[80930]: last_changed 2024-04-16T05:45:53.011672+0000 2024-04-16T05:45:54.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:54 smithi122 ceph-mon[80930]: created 2024-04-16T05:30:56.860193+0000 2024-04-16T05:45:54.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:54 smithi122 ceph-mon[80930]: min_mon_release 19 (squid) 2024-04-16T05:45:54.273 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:54 smithi122 ceph-mon[80930]: election_strategy: 1 2024-04-16T05:45:54.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:54 smithi122 ceph-mon[80930]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.smithi027 2024-04-16T05:45:54.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:54 smithi122 ceph-mon[80930]: 1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.smithi122 2024-04-16T05:45:54.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:54 smithi122 ceph-mon[80930]: fsmap cephfs:1 {0=cephfs.smithi027.ejoytk=up:active} 3 up:standby 2024-04-16T05:45:54.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:54 smithi122 ceph-mon[80930]: osdmap e49: 6 total, 6 up, 6 in 2024-04-16T05:45:54.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:54 smithi122 ceph-mon[80930]: mgrmap e38: smithi027.qgzypi(active, since 24s), standbys: smithi122.sgokkq 2024-04-16T05:45:54.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:54 smithi122 ceph-mon[80930]: overall HEALTH_OK 2024-04-16T05:45:54.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:54 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:54.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:54 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:54.274 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:54 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:45:54.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:54 smithi027 ceph-mon[93635]: pgmap v15: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:54.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:54 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi027"}]: dispatch 2024-04-16T05:45:54.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:54 smithi027 ceph-mon[93635]: mon.smithi027 calling monitor election 2024-04-16T05:45:54.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:54 smithi027 ceph-mon[93635]: mon.smithi122 calling monitor election 2024-04-16T05:45:54.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:54 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mon metadata", "id": "smithi122"}]: dispatch 2024-04-16T05:45:54.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:54 smithi027 ceph-mon[93635]: mon.smithi027 is new leader, mons smithi027,smithi122 in quorum (ranks 0,1) 2024-04-16T05:45:54.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:54 smithi027 ceph-mon[93635]: monmap epoch 3 2024-04-16T05:45:54.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:54 smithi027 ceph-mon[93635]: fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 2024-04-16T05:45:54.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:54 smithi027 ceph-mon[93635]: last_changed 2024-04-16T05:45:53.011672+0000 2024-04-16T05:45:54.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:54 smithi027 ceph-mon[93635]: created 2024-04-16T05:30:56.860193+0000 2024-04-16T05:45:54.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:54 smithi027 ceph-mon[93635]: min_mon_release 19 (squid) 2024-04-16T05:45:54.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:54 smithi027 ceph-mon[93635]: election_strategy: 1 2024-04-16T05:45:54.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:54 smithi027 ceph-mon[93635]: 0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.smithi027 2024-04-16T05:45:54.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:54 smithi027 ceph-mon[93635]: 1: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.smithi122 2024-04-16T05:45:54.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:54 smithi027 ceph-mon[93635]: fsmap cephfs:1 {0=cephfs.smithi027.ejoytk=up:active} 3 up:standby 2024-04-16T05:45:54.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:54 smithi027 ceph-mon[93635]: osdmap e49: 6 total, 6 up, 6 in 2024-04-16T05:45:54.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:54 smithi027 ceph-mon[93635]: mgrmap e38: smithi027.qgzypi(active, since 24s), standbys: smithi122.sgokkq 2024-04-16T05:45:54.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:54 smithi027 ceph-mon[93635]: overall HEALTH_OK 2024-04-16T05:45:54.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:54 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:54.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:54 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:54.309 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:54 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:45:56.298 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:56 smithi122 ceph-mon[80930]: pgmap v16: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:56.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:56 smithi027 ceph-mon[93635]: pgmap v16: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:58.660 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:58 smithi027 ceph-mon[93635]: pgmap v17: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:58.660 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:58.660 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:58.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:58 smithi122 ceph-mon[80930]: pgmap v17: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:45:58.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:58.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:45:59.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:45:59 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:45:59.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:45:59 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:46:00.678 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:00 smithi122 ceph-mon[80930]: pgmap v18: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:00.678 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:00.678 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:00.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:00 smithi027 ceph-mon[93635]: pgmap v18: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:00.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:00.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:02.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:02 smithi122 ceph-mon[80930]: pgmap v19: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:02.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:02 smithi027 ceph-mon[93635]: pgmap v19: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:02.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.811+0000 7f9cea200700 1 -- 172.21.15.27:0/3565223261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce4103420 msgr2=0x7f9ce4103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:02.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.811+0000 7f9cea200700 1 --2- 172.21.15.27:0/3565223261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce4103420 0x7f9ce4103830 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9cd4004940 tx=0x7f9cd4009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:02.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.811+0000 7f9cea200700 1 -- 172.21.15.27:0/3565223261 shutdown_connections 2024-04-16T05:46:02.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.811+0000 7f9cea200700 1 --2- 172.21.15.27:0/3565223261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce41046d0 0x7f9ce4104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:02.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.811+0000 7f9cea200700 1 --2- 172.21.15.27:0/3565223261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce4103420 0x7f9ce4103830 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:02.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.811+0000 7f9cea200700 1 -- 172.21.15.27:0/3565223261 >> 172.21.15.27:0/3565223261 conn(0x7f9ce4072640 msgr2=0x7f9ce4074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:02.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.811+0000 7f9cea200700 1 -- 172.21.15.27:0/3565223261 shutdown_connections 2024-04-16T05:46:02.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.811+0000 7f9cea200700 1 -- 172.21.15.27:0/3565223261 wait complete. 2024-04-16T05:46:02.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.812+0000 7f9cea200700 1 Processor -- start 2024-04-16T05:46:02.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.812+0000 7f9cea200700 1 -- start start 2024-04-16T05:46:02.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.813+0000 7f9cea200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce4103420 0x7f9ce4113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:02.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.813+0000 7f9cea200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce41046d0 0x7f9ce41144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:02.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.813+0000 7f9cea200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce4114aa0 con 0x7f9ce41046d0 2024-04-16T05:46:02.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.813+0000 7f9cea200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce4114c10 con 0x7f9ce4103420 2024-04-16T05:46:02.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.813+0000 7f9cdbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce41046d0 0x7f9ce41144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:02.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.813+0000 7f9cdbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce41046d0 0x7f9ce41144a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45470/0 (socket says 172.21.15.27:45470) 2024-04-16T05:46:02.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.813+0000 7f9cdbe00700 1 -- 172.21.15.27:0/674141065 learned_addr learned my addr 172.21.15.27:0/674141065 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:46:02.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.813+0000 7f9ce3400700 1 --2- 172.21.15.27:0/674141065 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce4103420 0x7f9ce4113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:02.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.814+0000 7f9cdbe00700 1 -- 172.21.15.27:0/674141065 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce4103420 msgr2=0x7f9ce4113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:02.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.814+0000 7f9cdbe00700 1 --2- 172.21.15.27:0/674141065 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce4103420 0x7f9ce4113f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:02.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.814+0000 7f9cdbe00700 1 -- 172.21.15.27:0/674141065 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd4005040 con 0x7f9ce41046d0 2024-04-16T05:46:02.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.814+0000 7f9cdbe00700 1 --2- 172.21.15.27:0/674141065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce41046d0 0x7f9ce41144a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc002a50 tx=0x7f9ccc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:02.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.814+0000 7f9cdb400700 1 -- 172.21.15.27:0/674141065 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ccc007cc0 con 0x7f9ce41046d0 2024-04-16T05:46:02.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.814+0000 7f9cdb400700 1 -- 172.21.15.27:0/674141065 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ccc00f040 con 0x7f9ce41046d0 2024-04-16T05:46:02.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.814+0000 7f9cdb400700 1 -- 172.21.15.27:0/674141065 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ccc009640 con 0x7f9ce41046d0 2024-04-16T05:46:02.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.814+0000 7f9cea200700 1 -- 172.21.15.27:0/674141065 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce4119670 con 0x7f9ce41046d0 2024-04-16T05:46:02.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.815+0000 7f9cea200700 1 -- 172.21.15.27:0/674141065 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce4119c30 con 0x7f9ce41046d0 2024-04-16T05:46:02.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.816+0000 7f9cea200700 1 -- 172.21.15.27:0/674141065 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce404c710 con 0x7f9ce41046d0 2024-04-16T05:46:02.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.817+0000 7f9cdb400700 1 -- 172.21.15.27:0/674141065 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f9ccc0097c0 con 0x7f9ce41046d0 2024-04-16T05:46:02.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.818+0000 7f9cdb400700 1 --2- 172.21.15.27:0/674141065 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9cc406ed30 0x7f9cc40711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:02.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.818+0000 7f9cdb400700 1 -- 172.21.15.27:0/674141065 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f9ccc08e850 con 0x7f9ce41046d0 2024-04-16T05:46:02.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.818+0000 7f9ce3400700 1 --2- 172.21.15.27:0/674141065 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9cc406ed30 0x7f9cc40711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:02.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.819+0000 7f9ce3400700 1 --2- 172.21.15.27:0/674141065 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9cc406ed30 0x7f9cc40711e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9cd400b3b0 tx=0x7f9cd4009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:02.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:02.824+0000 7f9cdb400700 1 -- 172.21.15.27:0/674141065 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9ccc05af10 con 0x7f9ce41046d0 2024-04-16T05:46:03.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.077+0000 7f9cea200700 1 -- 172.21.15.27:0/674141065 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ce4108a60 con 0x7f9cc406ed30 2024-04-16T05:46:03.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.080+0000 7f9cdb400700 1 -- 172.21.15.27:0/674141065 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f9ce4108a60 con 0x7f9cc406ed30 2024-04-16T05:46:03.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.083+0000 7f9cea200700 1 -- 172.21.15.27:0/674141065 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9cc406ed30 msgr2=0x7f9cc40711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:03.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.083+0000 7f9cea200700 1 --2- 172.21.15.27:0/674141065 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9cc406ed30 0x7f9cc40711e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9cd400b3b0 tx=0x7f9cd4009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:03.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.084+0000 7f9cea200700 1 -- 172.21.15.27:0/674141065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce41046d0 msgr2=0x7f9ce41144a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:03.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.084+0000 7f9cea200700 1 --2- 172.21.15.27:0/674141065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce41046d0 0x7f9ce41144a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc002a50 tx=0x7f9ccc0046b0 comp rx=0 tx=0).stop 2024-04-16T05:46:03.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.084+0000 7f9cea200700 1 -- 172.21.15.27:0/674141065 shutdown_connections 2024-04-16T05:46:03.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.084+0000 7f9cea200700 1 --2- 172.21.15.27:0/674141065 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9cc406ed30 0x7f9cc40711e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:03.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.084+0000 7f9cea200700 1 --2- 172.21.15.27:0/674141065 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9ce4103420 0x7f9ce4113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:03.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.084+0000 7f9cea200700 1 --2- 172.21.15.27:0/674141065 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9ce41046d0 0x7f9ce41144a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:03.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.084+0000 7f9cea200700 1 -- 172.21.15.27:0/674141065 >> 172.21.15.27:0/674141065 conn(0x7f9ce4072640 msgr2=0x7f9ce4073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:03.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.084+0000 7f9cea200700 1 -- 172.21.15.27:0/674141065 shutdown_connections 2024-04-16T05:46:03.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.084+0000 7f9cea200700 1 -- 172.21.15.27:0/674141065 wait complete. 2024-04-16T05:46:03.099 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:46:03.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.251+0000 7f6b39400700 1 -- 172.21.15.27:0/363766592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b340fcc20 msgr2=0x7f6b340fd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:03.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.251+0000 7f6b39400700 1 --2- 172.21.15.27:0/363766592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b340fcc20 0x7f6b340fd030 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6b24004940 tx=0x7f6b24009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:03.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.251+0000 7f6b39400700 1 -- 172.21.15.27:0/363766592 shutdown_connections 2024-04-16T05:46:03.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.251+0000 7f6b39400700 1 --2- 172.21.15.27:0/363766592 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b340fded0 0x7f6b340fe340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:03.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.251+0000 7f6b39400700 1 --2- 172.21.15.27:0/363766592 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b340fcc20 0x7f6b340fd030 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:03.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.251+0000 7f6b39400700 1 -- 172.21.15.27:0/363766592 >> 172.21.15.27:0/363766592 conn(0x7f6b340f83a0 msgr2=0x7f6b340fa7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:03.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.251+0000 7f6b39400700 1 -- 172.21.15.27:0/363766592 shutdown_connections 2024-04-16T05:46:03.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.252+0000 7f6b39400700 1 -- 172.21.15.27:0/363766592 wait complete. 2024-04-16T05:46:03.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.252+0000 7f6b39400700 1 Processor -- start 2024-04-16T05:46:03.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.253+0000 7f6b39400700 1 -- start start 2024-04-16T05:46:03.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.253+0000 7f6b39400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b340fcc20 0x7f6b3410fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.253+0000 7f6b39400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b340fded0 0x7f6b3410ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.253+0000 7f6b39400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b34110590 con 0x7f6b340fded0 2024-04-16T05:46:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.253+0000 7f6b39400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b34110de0 con 0x7f6b340fcc20 2024-04-16T05:46:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.253+0000 7f6b33400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b340fded0 0x7f6b3410ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.253+0000 7f6b33400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b340fded0 0x7f6b3410ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45490/0 (socket says 172.21.15.27:45490) 2024-04-16T05:46:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.253+0000 7f6b33400700 1 -- 172.21.15.27:0/2472830215 learned_addr learned my addr 172.21.15.27:0/2472830215 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:46:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.253+0000 7f6b33e00700 1 --2- 172.21.15.27:0/2472830215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b340fcc20 0x7f6b3410fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.254+0000 7f6b33400700 1 -- 172.21.15.27:0/2472830215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b340fcc20 msgr2=0x7f6b3410fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.254+0000 7f6b33400700 1 --2- 172.21.15.27:0/2472830215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b340fcc20 0x7f6b3410fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:03.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.254+0000 7f6b33400700 1 -- 172.21.15.27:0/2472830215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b24005040 con 0x7f6b340fded0 2024-04-16T05:46:03.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.254+0000 7f6b33400700 1 --2- 172.21.15.27:0/2472830215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b340fded0 0x7f6b3410ff90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6b20002a50 tx=0x7f6b200046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:03.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.254+0000 7f6b2be00700 1 -- 172.21.15.27:0/2472830215 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b20007cc0 con 0x7f6b340fded0 2024-04-16T05:46:03.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.254+0000 7f6b2be00700 1 -- 172.21.15.27:0/2472830215 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b2000f040 con 0x7f6b340fded0 2024-04-16T05:46:03.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.254+0000 7f6b39400700 1 -- 172.21.15.27:0/2472830215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b341110c0 con 0x7f6b340fded0 2024-04-16T05:46:03.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.254+0000 7f6b2be00700 1 -- 172.21.15.27:0/2472830215 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b20009640 con 0x7f6b340fded0 2024-04-16T05:46:03.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.255+0000 7f6b39400700 1 -- 172.21.15.27:0/2472830215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b34111600 con 0x7f6b340fded0 2024-04-16T05:46:03.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.256+0000 7f6b39400700 1 -- 172.21.15.27:0/2472830215 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b34101860 con 0x7f6b340fded0 2024-04-16T05:46:03.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.257+0000 7f6b2be00700 1 -- 172.21.15.27:0/2472830215 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f6b200097a0 con 0x7f6b340fded0 2024-04-16T05:46:03.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.258+0000 7f6b2be00700 1 --2- 172.21.15.27:0/2472830215 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6b1806ed30 0x7f6b180711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:03.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.258+0000 7f6b2be00700 1 -- 172.21.15.27:0/2472830215 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f6b2008e980 con 0x7f6b340fded0 2024-04-16T05:46:03.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.258+0000 7f6b33e00700 1 --2- 172.21.15.27:0/2472830215 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6b1806ed30 0x7f6b180711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:03.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.259+0000 7f6b33e00700 1 --2- 172.21.15.27:0/2472830215 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6b1806ed30 0x7f6b180711e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400b3b0 tx=0x7f6b24009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:03.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.263+0000 7f6b2be00700 1 -- 172.21.15.27:0/2472830215 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6b2005afc0 con 0x7f6b340fded0 2024-04-16T05:46:03.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 236, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 81837 2024-04-16T05:46:03.408 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 236, success, outsize: 120 2024-04-16T05:46:03.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 238, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 81921 2024-04-16T05:46:03.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 238, success, outsize: 120 2024-04-16T05:46:03.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 240, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82008 2024-04-16T05:46:03.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 240, success, outsize: 120 2024-04-16T05:46:03.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 242, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82101 2024-04-16T05:46:03.478 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 242, success, outsize: 120 2024-04-16T05:46:03.498 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 244, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82191 2024-04-16T05:46:03.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 244, success, outsize: 120 2024-04-16T05:46:03.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.516+0000 7f6b39400700 1 -- 172.21.15.27:0/2472830215 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b34111820 con 0x7f6b1806ed30 2024-04-16T05:46:03.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.519+0000 7f6b2be00700 1 -- 172.21.15.27:0/2472830215 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f6b34111820 con 0x7f6b1806ed30 2024-04-16T05:46:03.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 246, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82275 2024-04-16T05:46:03.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 246, success, outsize: 120 2024-04-16T05:46:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.521+0000 7f6b39400700 1 -- 172.21.15.27:0/2472830215 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6b1806ed30 msgr2=0x7f6b180711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.522+0000 7f6b39400700 1 --2- 172.21.15.27:0/2472830215 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6b1806ed30 0x7f6b180711e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6b2400b3b0 tx=0x7f6b24009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.522+0000 7f6b39400700 1 -- 172.21.15.27:0/2472830215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b340fded0 msgr2=0x7f6b3410ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.522+0000 7f6b39400700 1 --2- 172.21.15.27:0/2472830215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b340fded0 0x7f6b3410ff90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6b20002a50 tx=0x7f6b200046b0 comp rx=0 tx=0).stop 2024-04-16T05:46:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.522+0000 7f6b39400700 1 -- 172.21.15.27:0/2472830215 shutdown_connections 2024-04-16T05:46:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.522+0000 7f6b39400700 1 --2- 172.21.15.27:0/2472830215 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6b1806ed30 0x7f6b180711e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:03.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.522+0000 7f6b39400700 1 --2- 172.21.15.27:0/2472830215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6b340fcc20 0x7f6b3410fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.522+0000 7f6b39400700 1 --2- 172.21.15.27:0/2472830215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b340fded0 0x7f6b3410ff90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.522+0000 7f6b39400700 1 -- 172.21.15.27:0/2472830215 >> 172.21.15.27:0/2472830215 conn(0x7f6b340f83a0 msgr2=0x7f6b340f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.522+0000 7f6b39400700 1 -- 172.21.15.27:0/2472830215 shutdown_connections 2024-04-16T05:46:03.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.523+0000 7f6b39400700 1 -- 172.21.15.27:0/2472830215 wait complete. 2024-04-16T05:46:03.538 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 248, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82362 2024-04-16T05:46:03.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 248, success, outsize: 120 2024-04-16T05:46:03.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 250, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82455 2024-04-16T05:46:03.561 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 250, success, outsize: 120 2024-04-16T05:46:03.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 252, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82545 2024-04-16T05:46:03.584 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 252, success, outsize: 120 2024-04-16T05:46:03.605 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 254, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82629 2024-04-16T05:46:03.605 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 254, success, outsize: 120 2024-04-16T05:46:03.624 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 256, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82716 2024-04-16T05:46:03.625 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 256, success, outsize: 120 2024-04-16T05:46:03.645 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 258, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82809 2024-04-16T05:46:03.645 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 258, success, outsize: 120 2024-04-16T05:46:03.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.667+0000 7f3ccb000700 1 -- 172.21.15.27:0/2757827235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cc40fe0e0 msgr2=0x7f3cc40fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:03.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.667+0000 7f3ccb000700 1 --2- 172.21.15.27:0/2757827235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cc40fe0e0 0x7f3cc40fe550 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0004940 tx=0x7f3cb0009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:03.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.667+0000 7f3ccb000700 1 -- 172.21.15.27:0/2757827235 shutdown_connections 2024-04-16T05:46:03.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.667+0000 7f3ccb000700 1 --2- 172.21.15.27:0/2757827235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cc40fe0e0 0x7f3cc40fe550 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:03.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.667+0000 7f3ccb000700 1 --2- 172.21.15.27:0/2757827235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cc40fcef0 0x7f3cc40fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:03.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.667+0000 7f3ccb000700 1 -- 172.21.15.27:0/2757827235 >> 172.21.15.27:0/2757827235 conn(0x7f3cc40f8530 msgr2=0x7f3cc40fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:03.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.667+0000 7f3ccb000700 1 -- 172.21.15.27:0/2757827235 shutdown_connections 2024-04-16T05:46:03.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.668+0000 7f3ccb000700 1 -- 172.21.15.27:0/2757827235 wait complete. 2024-04-16T05:46:03.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.668+0000 7f3ccb000700 1 Processor -- start 2024-04-16T05:46:03.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.669+0000 7f3ccb000700 1 -- start start 2024-04-16T05:46:03.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.669+0000 7f3ccb000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cc40fcef0 0x7f3cc4071df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:03.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.669+0000 7f3ccb000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cc40fe0e0 0x7f3cc4070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:03.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.669+0000 7f3ccb000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc40708a0 con 0x7f3cc40fe0e0 2024-04-16T05:46:03.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.669+0000 7f3ccb000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc40709e0 con 0x7f3cc40fcef0 2024-04-16T05:46:03.668 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 260, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82899 2024-04-16T05:46:03.668 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 260, success, outsize: 120 2024-04-16T05:46:03.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.669+0000 7f3cc3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cc40fe0e0 0x7f3cc4070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:03.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.669+0000 7f3cc3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cc40fe0e0 0x7f3cc4070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45508/0 (socket says 172.21.15.27:45508) 2024-04-16T05:46:03.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.669+0000 7f3cc3400700 1 -- 172.21.15.27:0/4130295727 learned_addr learned my addr 172.21.15.27:0/4130295727 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:46:03.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.669+0000 7f3cc3e00700 1 --2- 172.21.15.27:0/4130295727 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cc40fcef0 0x7f3cc4071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:03.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.670+0000 7f3cc3400700 1 -- 172.21.15.27:0/4130295727 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cc40fcef0 msgr2=0x7f3cc4071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:03.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.670+0000 7f3cc3400700 1 --2- 172.21.15.27:0/4130295727 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cc40fcef0 0x7f3cc4071df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:03.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.670+0000 7f3cc3400700 1 -- 172.21.15.27:0/4130295727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb0005040 con 0x7f3cc40fe0e0 2024-04-16T05:46:03.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.670+0000 7f3cc3e00700 1 --2- 172.21.15.27:0/4130295727 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cc40fcef0 0x7f3cc4071df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T05:46:03.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.670+0000 7f3cc3400700 1 --2- 172.21.15.27:0/4130295727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cc40fe0e0 0x7f3cc4070310 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0009150 tx=0x7f3cb0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:03.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.670+0000 7f3cbbe00700 1 -- 172.21.15.27:0/4130295727 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cb000e040 con 0x7f3cc40fe0e0 2024-04-16T05:46:03.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.670+0000 7f3cbbe00700 1 -- 172.21.15.27:0/4130295727 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3cb000bbf0 con 0x7f3cc40fe0e0 2024-04-16T05:46:03.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.670+0000 7f3ccb000700 1 -- 172.21.15.27:0/4130295727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cc4070c60 con 0x7f3cc40fe0e0 2024-04-16T05:46:03.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.670+0000 7f3cbbe00700 1 -- 172.21.15.27:0/4130295727 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3cb001c630 con 0x7f3cc40fe0e0 2024-04-16T05:46:03.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.670+0000 7f3ccb000700 1 -- 172.21.15.27:0/4130295727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cc4115ac0 con 0x7f3cc40fe0e0 2024-04-16T05:46:03.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.671+0000 7f3ccb000700 1 -- 172.21.15.27:0/4130295727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cc405f5f0 con 0x7f3cc40fe0e0 2024-04-16T05:46:03.670 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.673+0000 7f3cbbe00700 1 -- 172.21.15.27:0/4130295727 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f3cb000bd60 con 0x7f3cc40fe0e0 2024-04-16T05:46:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.673+0000 7f3cbbe00700 1 --2- 172.21.15.27:0/4130295727 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3ca806ed20 0x7f3ca80711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:03.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.674+0000 7f3cbbe00700 1 -- 172.21.15.27:0/4130295727 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3cb008ff50 con 0x7f3cc40fe0e0 2024-04-16T05:46:03.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.674+0000 7f3cc3e00700 1 --2- 172.21.15.27:0/4130295727 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3ca806ed20 0x7f3ca80711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:03.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.674+0000 7f3cc3e00700 1 --2- 172.21.15.27:0/4130295727 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3ca806ed20 0x7f3ca80711d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4009fd0 tx=0x7f3cb4009f00 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:03.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.678+0000 7f3cbbe00700 1 -- 172.21.15.27:0/4130295727 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3cb0058b60 con 0x7f3cc40fe0e0 2024-04-16T05:46:03.688 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 262, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 82983 2024-04-16T05:46:03.688 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 262, success, outsize: 120 2024-04-16T05:46:03.707 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 264, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83070 2024-04-16T05:46:03.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 264, success, outsize: 120 2024-04-16T05:46:03.728 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 266, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83163 2024-04-16T05:46:03.728 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 266, success, outsize: 120 2024-04-16T05:46:03.748 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 268, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83253 2024-04-16T05:46:03.748 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 268, success, outsize: 120 2024-04-16T05:46:03.766 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 270, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83337 2024-04-16T05:46:03.766 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 270, success, outsize: 120 2024-04-16T05:46:03.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 272, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83424 2024-04-16T05:46:03.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 272, success, outsize: 120 2024-04-16T05:46:03.805 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 274, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83517 2024-04-16T05:46:03.805 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 274, success, outsize: 120 2024-04-16T05:46:03.826 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 276, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83607 2024-04-16T05:46:03.826 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 276, success, outsize: 120 2024-04-16T05:46:03.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 278, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83691 2024-04-16T05:46:03.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 278, success, outsize: 120 2024-04-16T05:46:03.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 280, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83778 2024-04-16T05:46:03.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 280, success, outsize: 120 2024-04-16T05:46:03.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 282, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83871 2024-04-16T05:46:03.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 282, success, outsize: 120 2024-04-16T05:46:03.912 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 284, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 83961 2024-04-16T05:46:03.912 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 284, success, outsize: 120 2024-04-16T05:46:03.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.917+0000 7f3ccb000700 1 -- 172.21.15.27:0/4130295727 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f3cc41024a0 con 0x7f3ca806ed20 2024-04-16T05:46:03.928 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 286, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84045 2024-04-16T05:46:03.928 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 286, success, outsize: 120 2024-04-16T05:46:03.944 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 288, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84132 2024-04-16T05:46:03.944 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 288, success, outsize: 120 2024-04-16T05:46:03.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.947+0000 7f3cbbe00700 1 -- 172.21.15.27:0/4130295727 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f3cc41024a0 con 0x7f3ca806ed20 2024-04-16T05:46:03.946 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:46:03.946 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (2m) 27s ago 13m 13.5M - 0.25.0 c8568f914cd2 4ed63a8a6b33 2024-04-16T05:46:03.946 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (13m) 27s ago 13m 6665k - 17.2.7-847-g45317f95 b333581ccdf5 f1acb6959c63 2024-04-16T05:46:03.946 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (11m) 6s ago 11m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 e81efc20e8c0 2024-04-16T05:46:03.946 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (2m) 27s ago 12m 75.3M - 9.4.7 954c08fa6188 f7ee2a1afd12 2024-04-16T05:46:03.946 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (5m) 27s ago 5m 23.4M - 17.2.7-847-g45317f95 b333581ccdf5 0824e2621be5 2024-04-16T05:46:03.946 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (5m) 27s ago 5m 13.1M - 17.2.7-847-g45317f95 b333581ccdf5 b332a72cc594 2024-04-16T05:46:03.947 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (5m) 6s ago 5m 13.3M - 17.2.7-847-g45317f95 b333581ccdf5 f5e36815ddfa 2024-04-16T05:46:03.947 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (5m) 6s ago 5m 13.2M - 17.2.7-847-g45317f95 b333581ccdf5 755a73c65a7f 2024-04-16T05:46:03.947 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:8443,9283,8765 running (116s) 27s ago 14m 478M - 19.0.0-2884-g218a6914 a4e4811815e4 b69a0b08e12b 2024-04-16T05:46:03.947 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283,8765 running (65s) 6s ago 11m 451M - 19.0.0-2884-g218a6914 a4e4811815e4 32e27bcd7e42 2024-04-16T05:46:03.947 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (45s) 27s ago 15m 29.4M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 299ca337e4ca 2024-04-16T05:46:03.947 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (12s) 6s ago 11m 21.5M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5db5f3037ed0 2024-04-16T05:46:03.947 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (2m) 27s ago 13m 7696k - 1.5.0 0da6a335fe13 5e29e29868b0 2024-04-16T05:46:03.947 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (2m) 6s ago 11m 7646k - 1.5.0 0da6a335fe13 a360922bc621 2024-04-16T05:46:03.947 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (10m) 27s ago 10m 47.7M 1557M 17.2.7-847-g45317f95 b333581ccdf5 15e049573bca 2024-04-16T05:46:03.947 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (10m) 27s ago 10m 47.3M 1557M 17.2.7-847-g45317f95 b333581ccdf5 e146295f6500 2024-04-16T05:46:03.947 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (9m) 27s ago 9m 49.0M 1557M 17.2.7-847-g45317f95 b333581ccdf5 6212f6a18fa4 2024-04-16T05:46:03.947 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (8m) 6s ago 8m 45.8M 2581M 17.2.7-847-g45317f95 b333581ccdf5 d6724ad05f9f 2024-04-16T05:46:03.947 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (8m) 6s ago 8m 44.6M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7a59908a3e10 2024-04-16T05:46:03.947 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (7m) 6s ago 7m 43.9M 2581M 17.2.7-847-g45317f95 b333581ccdf5 8e4b60163907 2024-04-16T05:46:03.948 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (73s) 27s ago 12m 44.4M - 2.43.0 a07b618ecd1d e70f55ba5ec0 2024-04-16T05:46:03.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.950+0000 7f3ccb000700 1 -- 172.21.15.27:0/4130295727 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3ca806ed20 msgr2=0x7f3ca80711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:03.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.950+0000 7f3ccb000700 1 --2- 172.21.15.27:0/4130295727 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3ca806ed20 0x7f3ca80711d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4009fd0 tx=0x7f3cb4009f00 comp rx=0 tx=0).stop 2024-04-16T05:46:03.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.950+0000 7f3ccb000700 1 -- 172.21.15.27:0/4130295727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cc40fe0e0 msgr2=0x7f3cc4070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:03.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.950+0000 7f3ccb000700 1 --2- 172.21.15.27:0/4130295727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cc40fe0e0 0x7f3cc4070310 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0009150 tx=0x7f3cb0009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:03.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.950+0000 7f3ccb000700 1 -- 172.21.15.27:0/4130295727 shutdown_connections 2024-04-16T05:46:03.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.950+0000 7f3ccb000700 1 --2- 172.21.15.27:0/4130295727 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3ca806ed20 0x7f3ca80711d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:03.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.950+0000 7f3ccb000700 1 --2- 172.21.15.27:0/4130295727 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3cc40fcef0 0x7f3cc4071df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:03.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.950+0000 7f3ccb000700 1 --2- 172.21.15.27:0/4130295727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3cc40fe0e0 0x7f3cc4070310 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:03.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.950+0000 7f3ccb000700 1 -- 172.21.15.27:0/4130295727 >> 172.21.15.27:0/4130295727 conn(0x7f3cc40f8530 msgr2=0x7f3cc40f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:03.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.951+0000 7f3ccb000700 1 -- 172.21.15.27:0/4130295727 shutdown_connections 2024-04-16T05:46:03.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:03.951+0000 7f3ccb000700 1 -- 172.21.15.27:0/4130295727 wait complete. 2024-04-16T05:46:03.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 290, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 84225 2024-04-16T05:46:03.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 290, success, outsize: 120 2024-04-16T05:46:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.108+0000 7fe81fe00700 1 -- 172.21.15.27:0/371000272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe818066f00 msgr2=0x7fe8180ff9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.108+0000 7fe81fe00700 1 --2- 172.21.15.27:0/371000272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe818066f00 0x7fe8180ff9e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe80c004940 tx=0x7fe80c009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.108+0000 7fe81fe00700 1 -- 172.21.15.27:0/371000272 shutdown_connections 2024-04-16T05:46:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.108+0000 7fe81fe00700 1 --2- 172.21.15.27:0/371000272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe8180fff20 0x7fe818102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.108+0000 7fe81fe00700 1 --2- 172.21.15.27:0/371000272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe818066f00 0x7fe8180ff9e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.108+0000 7fe81fe00700 1 -- 172.21.15.27:0/371000272 >> 172.21.15.27:0/371000272 conn(0x7fe8180f7450 msgr2=0x7fe8180f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.108+0000 7fe81fe00700 1 -- 172.21.15.27:0/371000272 shutdown_connections 2024-04-16T05:46:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.109+0000 7fe81fe00700 1 -- 172.21.15.27:0/371000272 wait complete. 2024-04-16T05:46:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.110+0000 7fe81fe00700 1 Processor -- start 2024-04-16T05:46:04.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.110+0000 7fe81fe00700 1 -- start start 2024-04-16T05:46:04.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.110+0000 7fe81fe00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe818066f00 0x7fe81810fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:04.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.110+0000 7fe81fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8180fff20 0x7fe8181100e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.110+0000 7fe81fe00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8181106e0 con 0x7fe8180fff20 2024-04-16T05:46:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.110+0000 7fe81fe00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe818110850 con 0x7fe818066f00 2024-04-16T05:46:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.110+0000 7fe81ca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8180fff20 0x7fe8181100e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.111+0000 7fe81ca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8180fff20 0x7fe8181100e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46826/0 (socket says 172.21.15.27:46826) 2024-04-16T05:46:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.111+0000 7fe81ca00700 1 -- 172.21.15.27:0/2750880072 learned_addr learned my addr 172.21.15.27:0/2750880072 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:46:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.111+0000 7fe81d400700 1 --2- 172.21.15.27:0/2750880072 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe818066f00 0x7fe81810fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:04.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.111+0000 7fe81ca00700 1 -- 172.21.15.27:0/2750880072 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe818066f00 msgr2=0x7fe81810fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:04.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.111+0000 7fe81ca00700 1 --2- 172.21.15.27:0/2750880072 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe818066f00 0x7fe81810fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:04.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.111+0000 7fe81ca00700 1 -- 172.21.15.27:0/2750880072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c005040 con 0x7fe8180fff20 2024-04-16T05:46:04.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.111+0000 7fe81ca00700 1 --2- 172.21.15.27:0/2750880072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8180fff20 0x7fe8181100e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe804002a50 tx=0x7fe8040046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:04.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.112+0000 7fe815600700 1 -- 172.21.15.27:0/2750880072 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe804007cc0 con 0x7fe8180fff20 2024-04-16T05:46:04.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.112+0000 7fe815600700 1 -- 172.21.15.27:0/2750880072 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe80400f040 con 0x7fe8180fff20 2024-04-16T05:46:04.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.112+0000 7fe815600700 1 -- 172.21.15.27:0/2750880072 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe804009640 con 0x7fe8180fff20 2024-04-16T05:46:04.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.112+0000 7fe81fe00700 1 -- 172.21.15.27:0/2750880072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8181152b0 con 0x7fe8180fff20 2024-04-16T05:46:04.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.112+0000 7fe81fe00700 1 -- 172.21.15.27:0/2750880072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe818115870 con 0x7fe8180fff20 2024-04-16T05:46:04.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.113+0000 7fe81fe00700 1 -- 172.21.15.27:0/2750880072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe81804c710 con 0x7fe8180fff20 2024-04-16T05:46:04.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.114+0000 7fe815600700 1 -- 172.21.15.27:0/2750880072 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fe8040097a0 con 0x7fe8180fff20 2024-04-16T05:46:04.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.115+0000 7fe815600700 1 --2- 172.21.15.27:0/2750880072 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe80006ed30 0x7fe8000711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:04.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.115+0000 7fe815600700 1 -- 172.21.15.27:0/2750880072 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fe80408e830 con 0x7fe8180fff20 2024-04-16T05:46:04.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.116+0000 7fe81d400700 1 --2- 172.21.15.27:0/2750880072 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe80006ed30 0x7fe8000711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:04.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.116+0000 7fe81d400700 1 --2- 172.21.15.27:0/2750880072 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe80006ed30 0x7fe8000711e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe80c00b970 tx=0x7fe80c009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:04.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.120+0000 7fe815600700 1 -- 172.21.15.27:0/2750880072 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe80405af70 con 0x7fe8180fff20 2024-04-16T05:46:04.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.451+0000 7fe81fe00700 1 -- 172.21.15.27:0/2750880072 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fe818115cf0 con 0x7fe8180fff20 2024-04-16T05:46:04.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.452+0000 7fe815600700 1 -- 172.21.15.27:0/2750880072 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+736 (secure 0 0 0) 0x7fe80408ce60 con 0x7fe8180fff20 2024-04-16T05:46:04.451 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:46:04.451 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:46:04.451 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:46:04.451 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:46:04.451 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:46:04.451 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:46:04.451 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:46:04.452 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:46:04.452 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T05:46:04.452 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:46:04.452 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:46:04.452 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T05:46:04.452 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:46:04.452 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:46:04.452 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 10, 2024-04-16T05:46:04.452 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T05:46:04.452 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:46:04.452 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:46:04.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.456+0000 7fe81fe00700 1 -- 172.21.15.27:0/2750880072 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe80006ed30 msgr2=0x7fe8000711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:04.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.456+0000 7fe81fe00700 1 --2- 172.21.15.27:0/2750880072 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe80006ed30 0x7fe8000711e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe80c00b970 tx=0x7fe80c009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:04.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.456+0000 7fe81fe00700 1 -- 172.21.15.27:0/2750880072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8180fff20 msgr2=0x7fe8181100e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:04.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.456+0000 7fe81fe00700 1 --2- 172.21.15.27:0/2750880072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8180fff20 0x7fe8181100e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe804002a50 tx=0x7fe8040046b0 comp rx=0 tx=0).stop 2024-04-16T05:46:04.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.456+0000 7fe81fe00700 1 -- 172.21.15.27:0/2750880072 shutdown_connections 2024-04-16T05:46:04.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.456+0000 7fe81fe00700 1 --2- 172.21.15.27:0/2750880072 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe80006ed30 0x7fe8000711e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:04.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.456+0000 7fe81fe00700 1 --2- 172.21.15.27:0/2750880072 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe818066f00 0x7fe81810fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:04.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.456+0000 7fe81fe00700 1 --2- 172.21.15.27:0/2750880072 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe8180fff20 0x7fe8181100e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:04.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.456+0000 7fe81fe00700 1 -- 172.21.15.27:0/2750880072 >> 172.21.15.27:0/2750880072 conn(0x7fe8180f7450 msgr2=0x7fe8180fbac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:04.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.456+0000 7fe81fe00700 1 -- 172.21.15.27:0/2750880072 shutdown_connections 2024-04-16T05:46:04.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.457+0000 7fe81fe00700 1 -- 172.21.15.27:0/2750880072 wait complete. 2024-04-16T05:46:04.621 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.623+0000 7ff919000700 1 -- 172.21.15.27:0/2228516679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9140662f0 msgr2=0x7ff9140ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:04.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.623+0000 7ff919000700 1 --2- 172.21.15.27:0/2228516679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9140662f0 0x7ff9140ff800 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff904004940 tx=0x7ff904009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:04.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.623+0000 7ff919000700 1 -- 172.21.15.27:0/2228516679 shutdown_connections 2024-04-16T05:46:04.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.623+0000 7ff919000700 1 --2- 172.21.15.27:0/2228516679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9140ffd40 0x7ff914102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:04.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.623+0000 7ff919000700 1 --2- 172.21.15.27:0/2228516679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9140662f0 0x7ff9140ff800 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:04.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.623+0000 7ff919000700 1 -- 172.21.15.27:0/2228516679 >> 172.21.15.27:0/2228516679 conn(0x7ff9140f7270 msgr2=0x7ff9140f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:04.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.623+0000 7ff919000700 1 -- 172.21.15.27:0/2228516679 shutdown_connections 2024-04-16T05:46:04.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.623+0000 7ff919000700 1 -- 172.21.15.27:0/2228516679 wait complete. 2024-04-16T05:46:04.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.625+0000 7ff919000700 1 Processor -- start 2024-04-16T05:46:04.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.625+0000 7ff919000700 1 -- start start 2024-04-16T05:46:04.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.626+0000 7ff919000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9140662f0 0x7ff9140fd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:04.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.626+0000 7ff919000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9140ffd40 0x7ff9140fb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:04.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.626+0000 7ff919000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9140fbc90 con 0x7ff9140ffd40 2024-04-16T05:46:04.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.626+0000 7ff919000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9140fbdd0 con 0x7ff9140662f0 2024-04-16T05:46:04.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.627+0000 7ff912a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9140ffd40 0x7ff9140fb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:04.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.627+0000 7ff912a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9140ffd40 0x7ff9140fb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46840/0 (socket says 172.21.15.27:46840) 2024-04-16T05:46:04.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.627+0000 7ff912a00700 1 -- 172.21.15.27:0/2069236183 learned_addr learned my addr 172.21.15.27:0/2069236183 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:46:04.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.627+0000 7ff913400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9140662f0 0x7ff9140fd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:04.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.627+0000 7ff912a00700 1 -- 172.21.15.27:0/2069236183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9140662f0 msgr2=0x7ff9140fd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:04.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.627+0000 7ff912a00700 1 --2- 172.21.15.27:0/2069236183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9140662f0 0x7ff9140fd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:04.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.627+0000 7ff912a00700 1 -- 172.21.15.27:0/2069236183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff900005040 con 0x7ff9140ffd40 2024-04-16T05:46:04.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.628+0000 7ff912a00700 1 --2- 172.21.15.27:0/2069236183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9140ffd40 0x7ff9140fb700 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff9000096d0 tx=0x7ff900009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:04.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.628+0000 7ff90b400700 1 -- 172.21.15.27:0/2069236183 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff90000e840 con 0x7ff9140ffd40 2024-04-16T05:46:04.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.628+0000 7ff90b400700 1 -- 172.21.15.27:0/2069236183 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff90000f040 con 0x7ff9140ffd40 2024-04-16T05:46:04.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.628+0000 7ff90b400700 1 -- 172.21.15.27:0/2069236183 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff900013dc0 con 0x7ff9140ffd40 2024-04-16T05:46:04.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.628+0000 7ff919000700 1 -- 172.21.15.27:0/2069236183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff904005040 con 0x7ff9140ffd40 2024-04-16T05:46:04.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.628+0000 7ff919000700 1 -- 172.21.15.27:0/2069236183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9140fc400 con 0x7ff9140ffd40 2024-04-16T05:46:04.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.630+0000 7ff919000700 1 -- 172.21.15.27:0/2069236183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff91405f5f0 con 0x7ff9140ffd40 2024-04-16T05:46:04.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.633+0000 7ff90b400700 1 -- 172.21.15.27:0/2069236183 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 92847+0+0 (secure 0 0 0) 0x7ff90000e9a0 con 0x7ff9140ffd40 2024-04-16T05:46:04.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.635+0000 7ff90b400700 1 --2- 172.21.15.27:0/2069236183 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff8f806ed20 0x7ff8f80711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:04.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.635+0000 7ff90b400700 1 -- 172.21.15.27:0/2069236183 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff90008e860 con 0x7ff9140ffd40 2024-04-16T05:46:04.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.635+0000 7ff913400700 1 --2- 172.21.15.27:0/2069236183 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff8f806ed20 0x7ff8f80711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:04.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.636+0000 7ff913400700 1 --2- 172.21.15.27:0/2069236183 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff8f806ed20 0x7ff8f80711d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff904013740 tx=0x7ff904009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:04.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.641+0000 7ff90b400700 1 -- 172.21.15.27:0/2069236183 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff9000573f0 con 0x7ff9140ffd40 2024-04-16T05:46:04.757 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:04 smithi122 ceph-mon[80930]: pgmap v20: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:04.757 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:04 smithi122 ceph-mon[80930]: from='client.34156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:46:04.757 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:04 smithi122 ceph-mon[80930]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:46:04.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:04 smithi027 ceph-mon[93635]: pgmap v20: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:04.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:04 smithi027 ceph-mon[93635]: from='client.34156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:46:04.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:04 smithi027 ceph-mon[93635]: from='client.34160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:46:04.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.932+0000 7ff919000700 1 -- 172.21.15.27:0/2069236183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff91404bed0 con 0x7ff9140ffd40 2024-04-16T05:46:04.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.933+0000 7ff90b400700 1 -- 172.21.15.27:0/2069236183 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1880 (secure 0 0 0) 0x7ff90005aab0 con 0x7ff9140ffd40 2024-04-16T05:46:04.931 INFO:teuthology.orchestra.run.smithi027.stdout:e16 2024-04-16T05:46:04.931 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:46:04.931 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:46:04.931 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:46:04.931 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:46:04.931 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:46:04.931 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:46:04.931 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 15 2024-04-16T05:46:04.931 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:46:04.932 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:46:04.932 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:40:32.701649+0000 2024-04-16T05:46:04.932 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:46:04.932 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:46:04.932 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:46:04.932 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:46:04.932 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:46:04.932 INFO:teuthology.orchestra.run.smithi027.stdout:max_xattr_size 65536 2024-04-16T05:46:04.932 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:46:04.932 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:46:04.932 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 46 2024-04-16T05:46:04.932 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:46:04.932 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:46:04.932 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:46:04.933 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=14462} 2024-04-16T05:46:04.933 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:46:04.933 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:46:04.933 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:46:04.933 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:46:04.933 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:46:04.933 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:46:04.933 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:46:04.933 INFO:teuthology.orchestra.run.smithi027.stdout:bal_rank_mask -1 2024-04-16T05:46:04.933 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:46:04.933 INFO:teuthology.orchestra.run.smithi027.stdout:qdb_cluster leader: 0 members: 2024-04-16T05:46:04.933 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:46:04.934 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:46:04.934 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:46:04.934 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:46:04.934 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:46:04.934 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{-1:14474} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2065690722,v1:172.21.15.27:6827/2065690722] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:46:04.934 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:24263} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:46:04.934 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:24273} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:46:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.937+0000 7ff919000700 1 -- 172.21.15.27:0/2069236183 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff8f806ed20 msgr2=0x7ff8f80711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.937+0000 7ff919000700 1 --2- 172.21.15.27:0/2069236183 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff8f806ed20 0x7ff8f80711d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff904013740 tx=0x7ff904009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.937+0000 7ff919000700 1 -- 172.21.15.27:0/2069236183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9140ffd40 msgr2=0x7ff9140fb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.937+0000 7ff919000700 1 --2- 172.21.15.27:0/2069236183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9140ffd40 0x7ff9140fb700 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff9000096d0 tx=0x7ff900009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.937+0000 7ff919000700 1 -- 172.21.15.27:0/2069236183 shutdown_connections 2024-04-16T05:46:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.937+0000 7ff919000700 1 --2- 172.21.15.27:0/2069236183 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff8f806ed20 0x7ff8f80711d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.937+0000 7ff919000700 1 --2- 172.21.15.27:0/2069236183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff9140662f0 0x7ff9140fd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.937+0000 7ff919000700 1 --2- 172.21.15.27:0/2069236183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff9140ffd40 0x7ff9140fb700 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:04.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.937+0000 7ff919000700 1 -- 172.21.15.27:0/2069236183 >> 172.21.15.27:0/2069236183 conn(0x7ff9140f7270 msgr2=0x7ff9140f7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:04.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.937+0000 7ff919000700 1 -- 172.21.15.27:0/2069236183 shutdown_connections 2024-04-16T05:46:04.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:04.937+0000 7ff919000700 1 -- 172.21.15.27:0/2069236183 wait complete. 2024-04-16T05:46:04.937 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2024-04-16T05:46:05.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.104+0000 7f191aa00700 1 -- 172.21.15.27:0/1561616374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19140fff70 msgr2=0x7f1914102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.104+0000 7f191aa00700 1 --2- 172.21.15.27:0/1561616374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19140fff70 0x7f1914102390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1900004940 tx=0x7f1900009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.104+0000 7f191aa00700 1 -- 172.21.15.27:0/1561616374 shutdown_connections 2024-04-16T05:46:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.104+0000 7f191aa00700 1 --2- 172.21.15.27:0/1561616374 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19140fff70 0x7f1914102390 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.104+0000 7f191aa00700 1 --2- 172.21.15.27:0/1561616374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19140662f0 0x7f19140ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.104+0000 7f191aa00700 1 -- 172.21.15.27:0/1561616374 >> 172.21.15.27:0/1561616374 conn(0x7f19140f74c0 msgr2=0x7f19140f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.104+0000 7f191aa00700 1 -- 172.21.15.27:0/1561616374 shutdown_connections 2024-04-16T05:46:05.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.104+0000 7f191aa00700 1 -- 172.21.15.27:0/1561616374 wait complete. 2024-04-16T05:46:05.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.105+0000 7f191aa00700 1 Processor -- start 2024-04-16T05:46:05.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.105+0000 7f191aa00700 1 -- start start 2024-04-16T05:46:05.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.106+0000 7f191aa00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19140662f0 0x7f191410d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:05.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.106+0000 7f191aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19140fff70 0x7f191410df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:05.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.106+0000 7f191aa00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f191410e510 con 0x7f19140fff70 2024-04-16T05:46:05.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.106+0000 7f191aa00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f191410ed60 con 0x7f19140662f0 2024-04-16T05:46:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.106+0000 7f1913400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19140fff70 0x7f191410df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.106+0000 7f1913400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19140fff70 0x7f191410df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46852/0 (socket says 172.21.15.27:46852) 2024-04-16T05:46:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.106+0000 7f1913400700 1 -- 172.21.15.27:0/283623014 learned_addr learned my addr 172.21.15.27:0/283623014 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:46:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.106+0000 7f1913e00700 1 --2- 172.21.15.27:0/283623014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19140662f0 0x7f191410d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.107+0000 7f1913400700 1 -- 172.21.15.27:0/283623014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19140662f0 msgr2=0x7f191410d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.107+0000 7f1913400700 1 --2- 172.21.15.27:0/283623014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19140662f0 0x7f191410d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:05.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.107+0000 7f1913400700 1 -- 172.21.15.27:0/283623014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1900005040 con 0x7f19140fff70 2024-04-16T05:46:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.107+0000 7f1913400700 1 --2- 172.21.15.27:0/283623014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19140fff70 0x7f191410df10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f190000b0d0 tx=0x7f1900009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.108+0000 7f190be00700 1 -- 172.21.15.27:0/283623014 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f190000e070 con 0x7f19140fff70 2024-04-16T05:46:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.108+0000 7f190be00700 1 -- 172.21.15.27:0/283623014 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f190000b410 con 0x7f19140fff70 2024-04-16T05:46:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.108+0000 7f190be00700 1 -- 172.21.15.27:0/283623014 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19000125e0 con 0x7f19140fff70 2024-04-16T05:46:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.108+0000 7f191aa00700 1 -- 172.21.15.27:0/283623014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f191410efe0 con 0x7f19140fff70 2024-04-16T05:46:05.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.108+0000 7f191aa00700 1 -- 172.21.15.27:0/283623014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f191410f4d0 con 0x7f19140fff70 2024-04-16T05:46:05.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.109+0000 7f191aa00700 1 -- 172.21.15.27:0/283623014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f191405f5f0 con 0x7f19140fff70 2024-04-16T05:46:05.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.110+0000 7f190be00700 1 -- 172.21.15.27:0/283623014 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f190000b580 con 0x7f19140fff70 2024-04-16T05:46:05.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.111+0000 7f190be00700 1 --2- 172.21.15.27:0/283623014 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f18f806ed20 0x7f18f80711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.111+0000 7f190be00700 1 -- 172.21.15.27:0/283623014 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f190008f770 con 0x7f19140fff70 2024-04-16T05:46:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.112+0000 7f1913e00700 1 --2- 172.21.15.27:0/283623014 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f18f806ed20 0x7f18f80711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:05.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.113+0000 7f1913e00700 1 --2- 172.21.15.27:0/283623014 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f18f806ed20 0x7f18f80711d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1904009b80 tx=0x7f190400b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:05.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.116+0000 7f190be00700 1 -- 172.21.15.27:0/283623014 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f190005bf70 con 0x7f19140fff70 2024-04-16T05:46:05.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.378+0000 7f191aa00700 1 -- 172.21.15.27:0/283623014 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f191410f870 con 0x7f18f806ed20 2024-04-16T05:46:05.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.380+0000 7f190be00700 1 -- 172.21.15.27:0/283623014 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f191410f870 con 0x7f18f806ed20 2024-04-16T05:46:05.378 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:46:05.378 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T05:46:05.378 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:46:05.378 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:46:05.379 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [ 2024-04-16T05:46:05.379 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr", 2024-04-16T05:46:05.379 INFO:teuthology.orchestra.run.smithi027.stdout: "mon" 2024-04-16T05:46:05.379 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-16T05:46:05.379 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "4/21 daemons upgraded", 2024-04-16T05:46:05.379 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Currently upgrading mon daemons", 2024-04-16T05:46:05.379 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:46:05.379 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:46:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.383+0000 7f191aa00700 1 -- 172.21.15.27:0/283623014 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f18f806ed20 msgr2=0x7f18f80711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.383+0000 7f191aa00700 1 --2- 172.21.15.27:0/283623014 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f18f806ed20 0x7f18f80711d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1904009b80 tx=0x7f190400b040 comp rx=0 tx=0).stop 2024-04-16T05:46:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.383+0000 7f191aa00700 1 -- 172.21.15.27:0/283623014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19140fff70 msgr2=0x7f191410df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.383+0000 7f191aa00700 1 --2- 172.21.15.27:0/283623014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19140fff70 0x7f191410df10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f190000b0d0 tx=0x7f1900009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.384+0000 7f191aa00700 1 -- 172.21.15.27:0/283623014 shutdown_connections 2024-04-16T05:46:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.384+0000 7f191aa00700 1 --2- 172.21.15.27:0/283623014 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f18f806ed20 0x7f18f80711d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.384+0000 7f191aa00700 1 --2- 172.21.15.27:0/283623014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19140662f0 0x7f191410d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.384+0000 7f191aa00700 1 --2- 172.21.15.27:0/283623014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19140fff70 0x7f191410df10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:05.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.384+0000 7f191aa00700 1 -- 172.21.15.27:0/283623014 >> 172.21.15.27:0/283623014 conn(0x7f19140f74c0 msgr2=0x7f19140f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.384+0000 7f191aa00700 1 -- 172.21.15.27:0/283623014 shutdown_connections 2024-04-16T05:46:05.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.384+0000 7f191aa00700 1 -- 172.21.15.27:0/283623014 wait complete. 2024-04-16T05:46:05.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.543+0000 7fcf16600700 1 -- 172.21.15.27:0/2581308863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf100fcdc0 msgr2=0x7fcf100fd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:05.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.543+0000 7fcf16600700 1 --2- 172.21.15.27:0/2581308863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf100fcdc0 0x7fcf100fd1d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcf00004940 tx=0x7fcf00009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:05.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.543+0000 7fcf16600700 1 -- 172.21.15.27:0/2581308863 shutdown_connections 2024-04-16T05:46:05.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.543+0000 7fcf16600700 1 --2- 172.21.15.27:0/2581308863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf100fd710 0x7fcf100fe070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:05.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.543+0000 7fcf16600700 1 --2- 172.21.15.27:0/2581308863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf100fcdc0 0x7fcf100fd1d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:05.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.543+0000 7fcf16600700 1 -- 172.21.15.27:0/2581308863 >> 172.21.15.27:0/2581308863 conn(0x7fcf100f8540 msgr2=0x7fcf100fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:05.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.543+0000 7fcf16600700 1 -- 172.21.15.27:0/2581308863 shutdown_connections 2024-04-16T05:46:05.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.544+0000 7fcf16600700 1 -- 172.21.15.27:0/2581308863 wait complete. 2024-04-16T05:46:05.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.544+0000 7fcf16600700 1 Processor -- start 2024-04-16T05:46:05.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.545+0000 7fcf16600700 1 -- start start 2024-04-16T05:46:05.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.545+0000 7fcf16600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf100fcdc0 0x7fcf1010da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:05.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.545+0000 7fcf16600700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf100fd710 0x7fcf1010dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:05.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.545+0000 7fcf16600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1010e5b0 con 0x7fcf100fcdc0 2024-04-16T05:46:05.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.545+0000 7fcf16600700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1010ee00 con 0x7fcf100fd710 2024-04-16T05:46:05.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.545+0000 7fcf0f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf100fcdc0 0x7fcf1010da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:05.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.546+0000 7fcf0f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf100fcdc0 0x7fcf1010da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46872/0 (socket says 172.21.15.27:46872) 2024-04-16T05:46:05.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.546+0000 7fcf0f400700 1 -- 172.21.15.27:0/2330486425 learned_addr learned my addr 172.21.15.27:0/2330486425 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:46:05.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.546+0000 7fcf0ea00700 1 --2- 172.21.15.27:0/2330486425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf100fd710 0x7fcf1010dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:05.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.546+0000 7fcf0f400700 1 -- 172.21.15.27:0/2330486425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf100fd710 msgr2=0x7fcf1010dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:05.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.546+0000 7fcf0f400700 1 --2- 172.21.15.27:0/2330486425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf100fd710 0x7fcf1010dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:05.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.546+0000 7fcf0f400700 1 -- 172.21.15.27:0/2330486425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf00005040 con 0x7fcf100fcdc0 2024-04-16T05:46:05.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.546+0000 7fcf0f400700 1 --2- 172.21.15.27:0/2330486425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf100fcdc0 0x7fcf1010da70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcf0000b970 tx=0x7fcf00009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:05.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.547+0000 7fcf07400700 1 -- 172.21.15.27:0/2330486425 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf0000e040 con 0x7fcf100fcdc0 2024-04-16T05:46:05.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.547+0000 7fcf07400700 1 -- 172.21.15.27:0/2330486425 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf00004290 con 0x7fcf100fcdc0 2024-04-16T05:46:05.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.547+0000 7fcf07400700 1 -- 172.21.15.27:0/2330486425 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf0001c630 con 0x7fcf100fcdc0 2024-04-16T05:46:05.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.547+0000 7fcf16600700 1 -- 172.21.15.27:0/2330486425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf1010f080 con 0x7fcf100fcdc0 2024-04-16T05:46:05.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.547+0000 7fcf16600700 1 -- 172.21.15.27:0/2330486425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1010f5c0 con 0x7fcf100fcdc0 2024-04-16T05:46:05.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.548+0000 7fcf16600700 1 -- 172.21.15.27:0/2330486425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1005f5f0 con 0x7fcf100fcdc0 2024-04-16T05:46:05.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.549+0000 7fcf07400700 1 -- 172.21.15.27:0/2330486425 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fcf0000bbf0 con 0x7fcf100fcdc0 2024-04-16T05:46:05.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.551+0000 7fcf07400700 1 --2- 172.21.15.27:0/2330486425 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fcef406ed20 0x7fcef40711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:05.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.551+0000 7fcf07400700 1 -- 172.21.15.27:0/2330486425 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fcf0008fe20 con 0x7fcf100fcdc0 2024-04-16T05:46:05.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.551+0000 7fcf0ea00700 1 --2- 172.21.15.27:0/2330486425 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fcef406ed20 0x7fcef40711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:05.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.552+0000 7fcf0ea00700 1 --2- 172.21.15.27:0/2330486425 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fcef406ed20 0x7fcef40711d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcf100fcb90 tx=0x7fcefc009ee0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:05.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.555+0000 7fcf07400700 1 -- 172.21.15.27:0/2330486425 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fcf0005c560 con 0x7fcf100fcdc0 2024-04-16T05:46:05.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:05 smithi122 ceph-mon[80930]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:46:05.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:05 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/2750880072' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:46:05.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:05 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:05.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:05 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:05.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:05 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:05.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:05 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:46:05.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:05 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:05.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:05 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/2069236183' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:46:05.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:05 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T05:46:05.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:05 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T05:46:05.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:05 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:05.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:05 smithi027 ceph-mon[93635]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:46:05.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:05 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/2750880072' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:46:05.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:05 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:05.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:05 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:05.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:05 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:05.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:05 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:46:05.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:05 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:05.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:05 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/2069236183' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:46:05.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:05 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T05:46:05.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:05 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T05:46:05.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:05 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:05.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.886+0000 7fcf16600700 1 -- 172.21.15.27:0/2330486425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fcf10105900 con 0x7fcf100fcdc0 2024-04-16T05:46:05.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.887+0000 7fcf07400700 1 -- 172.21.15.27:0/2330486425 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fcf00013e60 con 0x7fcf100fcdc0 2024-04-16T05:46:05.885 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:46:05.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.890+0000 7fcf16600700 1 -- 172.21.15.27:0/2330486425 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fcef406ed20 msgr2=0x7fcef40711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:05.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.890+0000 7fcf16600700 1 --2- 172.21.15.27:0/2330486425 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fcef406ed20 0x7fcef40711d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcf100fcb90 tx=0x7fcefc009ee0 comp rx=0 tx=0).stop 2024-04-16T05:46:05.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.890+0000 7fcf16600700 1 -- 172.21.15.27:0/2330486425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf100fcdc0 msgr2=0x7fcf1010da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:05.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.890+0000 7fcf16600700 1 --2- 172.21.15.27:0/2330486425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf100fcdc0 0x7fcf1010da70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcf0000b970 tx=0x7fcf00009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:05.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.890+0000 7fcf16600700 1 -- 172.21.15.27:0/2330486425 shutdown_connections 2024-04-16T05:46:05.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.890+0000 7fcf16600700 1 --2- 172.21.15.27:0/2330486425 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fcef406ed20 0x7fcef40711d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:05.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.890+0000 7fcf16600700 1 --2- 172.21.15.27:0/2330486425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcf100fcdc0 0x7fcf1010da70 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:05.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.890+0000 7fcf16600700 1 --2- 172.21.15.27:0/2330486425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcf100fd710 0x7fcf1010dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:05.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.890+0000 7fcf16600700 1 -- 172.21.15.27:0/2330486425 >> 172.21.15.27:0/2330486425 conn(0x7fcf100f8540 msgr2=0x7fcf100f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:05.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.890+0000 7fcf16600700 1 -- 172.21.15.27:0/2330486425 shutdown_connections 2024-04-16T05:46:05.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:05.890+0000 7fcf16600700 1 -- 172.21.15.27:0/2330486425 wait complete. 2024-04-16T05:46:06.662 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:06 smithi027 ceph-mon[93635]: pgmap v21: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:06.662 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:06 smithi027 ceph-mon[93635]: Reconfiguring mon.smithi027 (monmap changed)... 2024-04-16T05:46:06.662 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:06 smithi027 ceph-mon[93635]: Reconfiguring daemon mon.smithi027 on smithi027 2024-04-16T05:46:06.662 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:06 smithi027 ceph-mon[93635]: from='client.34176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:46:06.663 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:06 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/2330486425' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:46:06.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:06 smithi122 ceph-mon[80930]: pgmap v21: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:06.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:06 smithi122 ceph-mon[80930]: Reconfiguring mon.smithi027 (monmap changed)... 2024-04-16T05:46:06.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:06 smithi122 ceph-mon[80930]: Reconfiguring daemon mon.smithi027 on smithi027 2024-04-16T05:46:06.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:06 smithi122 ceph-mon[80930]: from='client.34176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:46:06.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:06 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/2330486425' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:46:08.744 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:08 smithi027 ceph-mon[93635]: pgmap v22: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:08.745 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:08 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:08.745 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:08 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:08.745 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:08 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi027.qgzypi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T05:46:08.745 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:08 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T05:46:08.745 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:08 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:08.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:08 smithi122 ceph-mon[80930]: pgmap v22: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:08.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:08 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:08.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:08 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:08.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:08 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi027.qgzypi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T05:46:08.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:08 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T05:46:08.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:08 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:09.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:09 smithi027 ceph-mon[93635]: Reconfiguring mgr.smithi027.qgzypi (monmap changed)... 2024-04-16T05:46:09.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:09 smithi027 ceph-mon[93635]: Reconfiguring daemon mgr.smithi027.qgzypi on smithi027 2024-04-16T05:46:09.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:09 smithi122 ceph-mon[80930]: Reconfiguring mgr.smithi027.qgzypi (monmap changed)... 2024-04-16T05:46:09.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:09 smithi122 ceph-mon[80930]: Reconfiguring daemon mgr.smithi027.qgzypi on smithi027 2024-04-16T05:46:10.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:10 smithi027 ceph-mon[93635]: pgmap v23: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:10.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:10 smithi122 ceph-mon[80930]: pgmap v23: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:11.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:11 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:11.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:11 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:11.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:11 smithi122 ceph-mon[80930]: Reconfiguring crash.smithi027 (monmap changed)... 2024-04-16T05:46:11.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:11 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi027", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T05:46:11.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:11 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:11.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:11 smithi122 ceph-mon[80930]: Reconfiguring daemon crash.smithi027 on smithi027 2024-04-16T05:46:11.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:11 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:11.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:11 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:11.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:11 smithi027 ceph-mon[93635]: Reconfiguring crash.smithi027 (monmap changed)... 2024-04-16T05:46:11.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:11 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi027", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T05:46:11.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:11 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:11.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:11 smithi027 ceph-mon[93635]: Reconfiguring daemon crash.smithi027 on smithi027 2024-04-16T05:46:12.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:12 smithi122 ceph-mon[80930]: pgmap v24: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:12.777 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:12 smithi027 ceph-mon[93635]: pgmap v24: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:14.153 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:14.153 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:14.154 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:13 smithi027 ceph-mon[93635]: pgmap v25: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:14.154 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:13 smithi027 ceph-mon[93635]: Reconfiguring osd.0 (monmap changed)... 2024-04-16T05:46:14.154 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T05:46:14.154 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:14.154 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:13 smithi027 ceph-mon[93635]: Reconfiguring daemon osd.0 on smithi027 2024-04-16T05:46:14.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:14.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:14.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:13 smithi122 ceph-mon[80930]: pgmap v25: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:14.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:13 smithi122 ceph-mon[80930]: Reconfiguring osd.0 (monmap changed)... 2024-04-16T05:46:14.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T05:46:14.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:14.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:13 smithi122 ceph-mon[80930]: Reconfiguring daemon osd.0 on smithi027 2024-04-16T05:46:15.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:46:15.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:46:16.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:15 smithi122 ceph-mon[80930]: pgmap v26: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:16.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:15 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:16.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:15 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:16.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:15 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T05:46:16.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:15 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:16.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:15 smithi027 ceph-mon[93635]: pgmap v26: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:16.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:15 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:16.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:15 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:16.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:15 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T05:46:16.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:15 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:17.014 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:16 smithi027 ceph-mon[93635]: Reconfiguring osd.1 (monmap changed)... 2024-04-16T05:46:17.014 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:16 smithi027 ceph-mon[93635]: Reconfiguring daemon osd.1 on smithi027 2024-04-16T05:46:17.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:16 smithi122 ceph-mon[80930]: Reconfiguring osd.1 (monmap changed)... 2024-04-16T05:46:17.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:16 smithi122 ceph-mon[80930]: Reconfiguring daemon osd.1 on smithi027 2024-04-16T05:46:18.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:17 smithi122 ceph-mon[80930]: pgmap v27: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:18.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:17 smithi027 ceph-mon[93635]: pgmap v27: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:19.613 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:19.613 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:19.613 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:19 smithi027 ceph-mon[93635]: Reconfiguring osd.2 (monmap changed)... 2024-04-16T05:46:19.614 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T05:46:19.614 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:19.614 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:19 smithi027 ceph-mon[93635]: Reconfiguring daemon osd.2 on smithi027 2024-04-16T05:46:19.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:19.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:19.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:19 smithi122 ceph-mon[80930]: Reconfiguring osd.2 (monmap changed)... 2024-04-16T05:46:19.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T05:46:19.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:19.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:19 smithi122 ceph-mon[80930]: Reconfiguring daemon osd.2 on smithi027 2024-04-16T05:46:20.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:20 smithi122 ceph-mon[80930]: pgmap v28: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:20.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:20 smithi027 ceph-mon[93635]: pgmap v28: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:22.454 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:22 smithi027 ceph-mon[93635]: pgmap v29: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:22.454 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:22 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:22.454 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:22 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:22.454 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:22 smithi027 ceph-mon[93635]: Reconfiguring mds.cephfs.smithi027.hkhpuq (monmap changed)... 2024-04-16T05:46:22.454 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:22 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.hkhpuq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:46:22.454 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:22 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:22.454 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:22 smithi027 ceph-mon[93635]: Reconfiguring daemon mds.cephfs.smithi027.hkhpuq on smithi027 2024-04-16T05:46:22.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:22 smithi122 ceph-mon[80930]: pgmap v29: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:22.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:22 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:22.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:22 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:22.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:22 smithi122 ceph-mon[80930]: Reconfiguring mds.cephfs.smithi027.hkhpuq (monmap changed)... 2024-04-16T05:46:22.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:22 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.hkhpuq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:46:22.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:22 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:22.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:22 smithi122 ceph-mon[80930]: Reconfiguring daemon mds.cephfs.smithi027.hkhpuq on smithi027 2024-04-16T05:46:24.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:24 smithi122 ceph-mon[80930]: pgmap v30: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:24.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:24.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:24.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.ejoytk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:46:24.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:24.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:24 smithi027 ceph-mon[93635]: pgmap v30: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:24.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:24.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:24.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.ejoytk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:46:24.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:25.505 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:25 smithi027 ceph-mon[93635]: Reconfiguring mds.cephfs.smithi027.ejoytk (monmap changed)... 2024-04-16T05:46:25.505 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:25 smithi027 ceph-mon[93635]: Reconfiguring daemon mds.cephfs.smithi027.ejoytk on smithi027 2024-04-16T05:46:25.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:25 smithi122 ceph-mon[80930]: Reconfiguring mds.cephfs.smithi027.ejoytk (monmap changed)... 2024-04-16T05:46:25.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:25 smithi122 ceph-mon[80930]: Reconfiguring daemon mds.cephfs.smithi027.ejoytk on smithi027 2024-04-16T05:46:26.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:26 smithi122 ceph-mon[80930]: pgmap v31: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:26.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:26 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:26.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:26 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:26.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:26 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T05:46:26.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:26 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:26.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:26 smithi027 ceph-mon[93635]: pgmap v31: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:26.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:26 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:26.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:26 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:26.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:26 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T05:46:26.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:26 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:27.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:27 smithi122 ceph-mon[80930]: Reconfiguring crash.smithi122 (monmap changed)... 2024-04-16T05:46:27.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:27 smithi122 ceph-mon[80930]: Reconfiguring daemon crash.smithi122 on smithi122 2024-04-16T05:46:27.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:27 smithi027 ceph-mon[93635]: Reconfiguring crash.smithi122 (monmap changed)... 2024-04-16T05:46:27.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:27 smithi027 ceph-mon[93635]: Reconfiguring daemon crash.smithi122 on smithi122 2024-04-16T05:46:28.437 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:28 smithi027 ceph-mon[93635]: pgmap v32: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:28.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:28 smithi122 ceph-mon[80930]: pgmap v32: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:29.467 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:29.467 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:29.467 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:29 smithi122 ceph-mon[80930]: Reconfiguring mgr.smithi122.sgokkq (monmap changed)... 2024-04-16T05:46:29.467 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.sgokkq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T05:46:29.467 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T05:46:29.467 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:29.468 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:29 smithi122 ceph-mon[80930]: Reconfiguring daemon mgr.smithi122.sgokkq on smithi122 2024-04-16T05:46:29.468 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:46:29.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:29.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:29.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:29 smithi027 ceph-mon[93635]: Reconfiguring mgr.smithi122.sgokkq (monmap changed)... 2024-04-16T05:46:29.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi122.sgokkq", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-16T05:46:29.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-16T05:46:29.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:29.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:29 smithi027 ceph-mon[93635]: Reconfiguring daemon mgr.smithi122.sgokkq on smithi122 2024-04-16T05:46:29.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:46:30.470 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:30 smithi122 ceph-mon[80930]: pgmap v33: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:30.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:30 smithi027 ceph-mon[93635]: pgmap v33: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:31.891 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:31 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:31.891 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:31 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:31.891 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:31 smithi122 ceph-mon[80930]: Reconfiguring mon.smithi122 (monmap changed)... 2024-04-16T05:46:31.891 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:31 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T05:46:31.891 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:31 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T05:46:31.891 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:31 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:31.891 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:31 smithi122 ceph-mon[80930]: Reconfiguring daemon mon.smithi122 on smithi122 2024-04-16T05:46:31.891 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:31 smithi122 ceph-mon[80930]: pgmap v34: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:31 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:31 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:31 smithi027 ceph-mon[93635]: Reconfiguring mon.smithi122 (monmap changed)... 2024-04-16T05:46:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:31 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-16T05:46:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:31 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-16T05:46:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:31 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:31 smithi027 ceph-mon[93635]: Reconfiguring daemon mon.smithi122 on smithi122 2024-04-16T05:46:32.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:31 smithi027 ceph-mon[93635]: pgmap v34: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:34.110 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:33 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:34.110 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:33 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:34.110 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:33 smithi122 ceph-mon[80930]: Reconfiguring osd.3 (monmap changed)... 2024-04-16T05:46:34.110 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:33 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T05:46:34.110 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:33 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:34.111 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:33 smithi122 ceph-mon[80930]: Reconfiguring daemon osd.3 on smithi122 2024-04-16T05:46:34.111 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:33 smithi122 ceph-mon[80930]: pgmap v35: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:34.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:33 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:34.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:33 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:34.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:33 smithi027 ceph-mon[93635]: Reconfiguring osd.3 (monmap changed)... 2024-04-16T05:46:34.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:33 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T05:46:34.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:33 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:34.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:33 smithi027 ceph-mon[93635]: Reconfiguring daemon osd.3 on smithi122 2024-04-16T05:46:34.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:33 smithi027 ceph-mon[93635]: pgmap v35: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:36.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.065+0000 7f3e70e00700 1 -- 172.21.15.27:0/2793770362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e6c103420 msgr2=0x7f3e6c103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.065+0000 7f3e70e00700 1 --2- 172.21.15.27:0/2793770362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e6c103420 0x7f3e6c103830 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c004940 tx=0x7f3e5c009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.065+0000 7f3e70e00700 1 -- 172.21.15.27:0/2793770362 shutdown_connections 2024-04-16T05:46:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.065+0000 7f3e70e00700 1 --2- 172.21.15.27:0/2793770362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3e6c1046d0 0x7f3e6c104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.065+0000 7f3e70e00700 1 --2- 172.21.15.27:0/2793770362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e6c103420 0x7f3e6c103830 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.065+0000 7f3e70e00700 1 -- 172.21.15.27:0/2793770362 >> 172.21.15.27:0/2793770362 conn(0x7f3e6c072640 msgr2=0x7f3e6c074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.065+0000 7f3e70e00700 1 -- 172.21.15.27:0/2793770362 shutdown_connections 2024-04-16T05:46:36.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.065+0000 7f3e70e00700 1 -- 172.21.15.27:0/2793770362 wait complete. 2024-04-16T05:46:36.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.066+0000 7f3e70e00700 1 Processor -- start 2024-04-16T05:46:36.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.066+0000 7f3e70e00700 1 -- start start 2024-04-16T05:46:36.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.067+0000 7f3e70e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e6c103420 0x7f3e6c113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:36.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.067+0000 7f3e70e00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3e6c1046d0 0x7f3e6c114420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.067+0000 7f3e70e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c114a20 con 0x7f3e6c103420 2024-04-16T05:46:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.067+0000 7f3e70e00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c114b90 con 0x7f3e6c1046d0 2024-04-16T05:46:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.067+0000 7f3e69e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e6c103420 0x7f3e6c113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.067+0000 7f3e69e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e6c103420 0x7f3e6c113ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33950/0 (socket says 172.21.15.27:33950) 2024-04-16T05:46:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.067+0000 7f3e69e00700 1 -- 172.21.15.27:0/394846611 learned_addr learned my addr 172.21.15.27:0/394846611 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:46:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.067+0000 7f3e69400700 1 --2- 172.21.15.27:0/394846611 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3e6c1046d0 0x7f3e6c114420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.068+0000 7f3e69e00700 1 -- 172.21.15.27:0/394846611 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3e6c1046d0 msgr2=0x7f3e6c114420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.068+0000 7f3e69e00700 1 --2- 172.21.15.27:0/394846611 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3e6c1046d0 0x7f3e6c114420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:36.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.068+0000 7f3e69e00700 1 -- 172.21.15.27:0/394846611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5c005040 con 0x7f3e6c103420 2024-04-16T05:46:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.068+0000 7f3e69e00700 1 --2- 172.21.15.27:0/394846611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e6c103420 0x7f3e6c113ee0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c00b0d0 tx=0x7f3e5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:36.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.068+0000 7f3e62000700 1 -- 172.21.15.27:0/394846611 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e5c00e070 con 0x7f3e6c103420 2024-04-16T05:46:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.068+0000 7f3e62000700 1 -- 172.21.15.27:0/394846611 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e5c00b410 con 0x7f3e6c103420 2024-04-16T05:46:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.068+0000 7f3e70e00700 1 -- 172.21.15.27:0/394846611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e6c119590 con 0x7f3e6c103420 2024-04-16T05:46:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.068+0000 7f3e62000700 1 -- 172.21.15.27:0/394846611 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e5c0125e0 con 0x7f3e6c103420 2024-04-16T05:46:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.068+0000 7f3e70e00700 1 -- 172.21.15.27:0/394846611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e6c119b50 con 0x7f3e6c103420 2024-04-16T05:46:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.069+0000 7f3e70e00700 1 -- 172.21.15.27:0/394846611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e6c04c710 con 0x7f3e6c103420 2024-04-16T05:46:36.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.071+0000 7f3e62000700 1 -- 172.21.15.27:0/394846611 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f3e5c00b580 con 0x7f3e6c103420 2024-04-16T05:46:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.072+0000 7f3e62000700 1 --2- 172.21.15.27:0/394846611 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3e5006ed30 0x7f3e500711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.072+0000 7f3e62000700 1 -- 172.21.15.27:0/394846611 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3e5c08f7e0 con 0x7f3e6c103420 2024-04-16T05:46:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.072+0000 7f3e69400700 1 --2- 172.21.15.27:0/394846611 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3e5006ed30 0x7f3e500711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:36.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.073+0000 7f3e69400700 1 --2- 172.21.15.27:0/394846611 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3e5006ed30 0x7f3e500711e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3e58009b80 tx=0x7f3e5800b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:36.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.077+0000 7f3e62000700 1 -- 172.21.15.27:0/394846611 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3e5c05bf20 con 0x7f3e6c103420 2024-04-16T05:46:36.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.328+0000 7f3e70e00700 1 -- 172.21.15.27:0/394846611 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3e6c1089a0 con 0x7f3e5006ed30 2024-04-16T05:46:36.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:36 smithi027 ceph-mon[93635]: pgmap v36: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:36.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:36 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:36.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:36 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:36.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:36 smithi027 ceph-mon[93635]: Reconfiguring osd.4 (monmap changed)... 2024-04-16T05:46:36.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:36 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T05:46:36.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:36 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:36.327 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:36 smithi027 ceph-mon[93635]: Reconfiguring daemon osd.4 on smithi122 2024-04-16T05:46:36.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.330+0000 7f3e62000700 1 -- 172.21.15.27:0/394846611 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f3e6c1089a0 con 0x7f3e5006ed30 2024-04-16T05:46:36.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.333+0000 7f3e70e00700 1 -- 172.21.15.27:0/394846611 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3e5006ed30 msgr2=0x7f3e500711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:36.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.333+0000 7f3e70e00700 1 --2- 172.21.15.27:0/394846611 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3e5006ed30 0x7f3e500711e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3e58009b80 tx=0x7f3e5800b040 comp rx=0 tx=0).stop 2024-04-16T05:46:36.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.333+0000 7f3e70e00700 1 -- 172.21.15.27:0/394846611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e6c103420 msgr2=0x7f3e6c113ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:36.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.333+0000 7f3e70e00700 1 --2- 172.21.15.27:0/394846611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e6c103420 0x7f3e6c113ee0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c00b0d0 tx=0x7f3e5c009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:36.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.333+0000 7f3e70e00700 1 -- 172.21.15.27:0/394846611 shutdown_connections 2024-04-16T05:46:36.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.333+0000 7f3e70e00700 1 --2- 172.21.15.27:0/394846611 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3e5006ed30 0x7f3e500711e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:36.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.333+0000 7f3e70e00700 1 --2- 172.21.15.27:0/394846611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e6c103420 0x7f3e6c113ee0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:36.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.333+0000 7f3e70e00700 1 --2- 172.21.15.27:0/394846611 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3e6c1046d0 0x7f3e6c114420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:36.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.333+0000 7f3e70e00700 1 -- 172.21.15.27:0/394846611 >> 172.21.15.27:0/394846611 conn(0x7f3e6c072640 msgr2=0x7f3e6c073cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:36.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.333+0000 7f3e70e00700 1 -- 172.21.15.27:0/394846611 shutdown_connections 2024-04-16T05:46:36.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.334+0000 7f3e70e00700 1 -- 172.21.15.27:0/394846611 wait complete. 2024-04-16T05:46:36.349 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:46:36.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.495+0000 7fd8da200700 1 -- 172.21.15.27:0/1564923578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8d41046b0 msgr2=0x7fd8d4104b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:36.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.495+0000 7fd8da200700 1 --2- 172.21.15.27:0/1564923578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8d41046b0 0x7fd8d4104b20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc004940 tx=0x7fd8bc009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:36.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.495+0000 7fd8da200700 1 -- 172.21.15.27:0/1564923578 shutdown_connections 2024-04-16T05:46:36.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.495+0000 7fd8da200700 1 --2- 172.21.15.27:0/1564923578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8d41046b0 0x7fd8d4104b20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:36.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.495+0000 7fd8da200700 1 --2- 172.21.15.27:0/1564923578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8d4103400 0x7fd8d4103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:36.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.495+0000 7fd8da200700 1 -- 172.21.15.27:0/1564923578 >> 172.21.15.27:0/1564923578 conn(0x7fd8d4072590 msgr2=0x7fd8d40749c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:36.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.495+0000 7fd8da200700 1 -- 172.21.15.27:0/1564923578 shutdown_connections 2024-04-16T05:46:36.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.495+0000 7fd8da200700 1 -- 172.21.15.27:0/1564923578 wait complete. 2024-04-16T05:46:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.496+0000 7fd8da200700 1 Processor -- start 2024-04-16T05:46:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.496+0000 7fd8da200700 1 -- start start 2024-04-16T05:46:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.496+0000 7fd8da200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8d4103400 0x7fd8d4114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.496+0000 7fd8da200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8d41046b0 0x7fd8d4114560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.496+0000 7fd8da200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d4114b60 con 0x7fd8d41046b0 2024-04-16T05:46:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.496+0000 7fd8da200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d41153b0 con 0x7fd8d4103400 2024-04-16T05:46:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.497+0000 7fd8caa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8d41046b0 0x7fd8d4114560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.497+0000 7fd8caa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8d41046b0 0x7fd8d4114560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33970/0 (socket says 172.21.15.27:33970) 2024-04-16T05:46:36.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.497+0000 7fd8caa00700 1 -- 172.21.15.27:0/2262665369 learned_addr learned my addr 172.21.15.27:0/2262665369 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:46:36.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.497+0000 7fd8caa00700 1 -- 172.21.15.27:0/2262665369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8d4103400 msgr2=0x7fd8d4114020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:36.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.497+0000 7fd8d3400700 1 --2- 172.21.15.27:0/2262665369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8d4103400 0x7fd8d4114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:36.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.497+0000 7fd8caa00700 1 --2- 172.21.15.27:0/2262665369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8d4103400 0x7fd8d4114020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:36.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.497+0000 7fd8caa00700 1 -- 172.21.15.27:0/2262665369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8bc005040 con 0x7fd8d41046b0 2024-04-16T05:46:36.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.497+0000 7fd8d3400700 1 --2- 172.21.15.27:0/2262665369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8d4103400 0x7fd8d4114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:46:36.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.498+0000 7fd8caa00700 1 --2- 172.21.15.27:0/2262665369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8d41046b0 0x7fd8d4114560 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00b3b0 tx=0x7fd8bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:36.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.498+0000 7fd8cbe00700 1 -- 172.21.15.27:0/2262665369 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8bc00e040 con 0x7fd8d41046b0 2024-04-16T05:46:36.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.498+0000 7fd8cbe00700 1 -- 172.21.15.27:0/2262665369 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd8bc004290 con 0x7fd8d41046b0 2024-04-16T05:46:36.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.498+0000 7fd8da200700 1 -- 172.21.15.27:0/2262665369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8d4115630 con 0x7fd8d41046b0 2024-04-16T05:46:36.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.498+0000 7fd8cbe00700 1 -- 172.21.15.27:0/2262665369 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8bc01c630 con 0x7fd8d41046b0 2024-04-16T05:46:36.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.498+0000 7fd8da200700 1 -- 172.21.15.27:0/2262665369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8d4115b70 con 0x7fd8d41046b0 2024-04-16T05:46:36.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.499+0000 7fd8da200700 1 -- 172.21.15.27:0/2262665369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8d405f5f0 con 0x7fd8d41046b0 2024-04-16T05:46:36.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.500+0000 7fd8cbe00700 1 -- 172.21.15.27:0/2262665369 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 92847+0+0 (secure 0 0 0) 0x7fd8bc00bbf0 con 0x7fd8d41046b0 2024-04-16T05:46:36.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.501+0000 7fd8cbe00700 1 --2- 172.21.15.27:0/2262665369 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd8b806ed20 0x7fd8b80711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.502+0000 7fd8cbe00700 1 -- 172.21.15.27:0/2262665369 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fd8bc08fe30 con 0x7fd8d41046b0 2024-04-16T05:46:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.502+0000 7fd8d3400700 1 --2- 172.21.15.27:0/2262665369 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd8b806ed20 0x7fd8b80711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:36.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.503+0000 7fd8d3400700 1 --2- 172.21.15.27:0/2262665369 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd8b806ed20 0x7fd8b80711d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4004550 tx=0x7fd8c4009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:36.505 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.507+0000 7fd8cbe00700 1 -- 172.21.15.27:0/2262665369 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd8bc05c630 con 0x7fd8d41046b0 2024-04-16T05:46:36.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:36 smithi122 ceph-mon[80930]: pgmap v36: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:36.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:36 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:36.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:36 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:36.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:36 smithi122 ceph-mon[80930]: Reconfiguring osd.4 (monmap changed)... 2024-04-16T05:46:36.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:36 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T05:46:36.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:36 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:36.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:36 smithi122 ceph-mon[80930]: Reconfiguring daemon osd.4 on smithi122 2024-04-16T05:46:36.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.762+0000 7fd8da200700 1 -- 172.21.15.27:0/2262665369 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8d4115f50 con 0x7fd8b806ed20 2024-04-16T05:46:36.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.764+0000 7fd8cbe00700 1 -- 172.21.15.27:0/2262665369 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7fd8d4115f50 con 0x7fd8b806ed20 2024-04-16T05:46:36.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.767+0000 7fd8da200700 1 -- 172.21.15.27:0/2262665369 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd8b806ed20 msgr2=0x7fd8b80711d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:36.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.767+0000 7fd8da200700 1 --2- 172.21.15.27:0/2262665369 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd8b806ed20 0x7fd8b80711d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4004550 tx=0x7fd8c4009040 comp rx=0 tx=0).stop 2024-04-16T05:46:36.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.767+0000 7fd8da200700 1 -- 172.21.15.27:0/2262665369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8d41046b0 msgr2=0x7fd8d4114560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:36.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.767+0000 7fd8da200700 1 --2- 172.21.15.27:0/2262665369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8d41046b0 0x7fd8d4114560 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00b3b0 tx=0x7fd8bc009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:36.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.767+0000 7fd8da200700 1 -- 172.21.15.27:0/2262665369 shutdown_connections 2024-04-16T05:46:36.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.767+0000 7fd8da200700 1 --2- 172.21.15.27:0/2262665369 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd8b806ed20 0x7fd8b80711d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:36.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.767+0000 7fd8da200700 1 --2- 172.21.15.27:0/2262665369 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8d4103400 0x7fd8d4114020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:36.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.768+0000 7fd8da200700 1 --2- 172.21.15.27:0/2262665369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8d41046b0 0x7fd8d4114560 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:36.766 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.768+0000 7fd8da200700 1 -- 172.21.15.27:0/2262665369 >> 172.21.15.27:0/2262665369 conn(0x7fd8d4072590 msgr2=0x7fd8d4073c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:36.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.768+0000 7fd8da200700 1 -- 172.21.15.27:0/2262665369 shutdown_connections 2024-04-16T05:46:36.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.768+0000 7fd8da200700 1 -- 172.21.15.27:0/2262665369 wait complete. 2024-04-16T05:46:36.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.925+0000 7f6281e00700 1 -- 172.21.15.27:0/145220492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f627c103420 msgr2=0x7f627c103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:36.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.925+0000 7f6281e00700 1 --2- 172.21.15.27:0/145220492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f627c103420 0x7f627c103830 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f626c004940 tx=0x7f626c009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:36.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.925+0000 7f6281e00700 1 -- 172.21.15.27:0/145220492 shutdown_connections 2024-04-16T05:46:36.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.925+0000 7f6281e00700 1 --2- 172.21.15.27:0/145220492 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f627c1046d0 0x7f627c104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:36.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.925+0000 7f6281e00700 1 --2- 172.21.15.27:0/145220492 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f627c103420 0x7f627c103830 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:36.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.925+0000 7f6281e00700 1 -- 172.21.15.27:0/145220492 >> 172.21.15.27:0/145220492 conn(0x7f627c072640 msgr2=0x7f627c074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:36.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.925+0000 7f6281e00700 1 -- 172.21.15.27:0/145220492 shutdown_connections 2024-04-16T05:46:36.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.925+0000 7f6281e00700 1 -- 172.21.15.27:0/145220492 wait complete. 2024-04-16T05:46:36.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.926+0000 7f6281e00700 1 Processor -- start 2024-04-16T05:46:36.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.926+0000 7f6281e00700 1 -- start start 2024-04-16T05:46:36.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.927+0000 7f6281e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f627c103420 0x7f627c113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:36.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.927+0000 7f6281e00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f627c1046d0 0x7f627c1144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:36.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.927+0000 7f6281e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627c114aa0 con 0x7f627c103420 2024-04-16T05:46:36.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.927+0000 7f6281e00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627c114c10 con 0x7f627c1046d0 2024-04-16T05:46:36.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.927+0000 7f627aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f627c103420 0x7f627c113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:36.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.927+0000 7f627aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f627c103420 0x7f627c113f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33990/0 (socket says 172.21.15.27:33990) 2024-04-16T05:46:36.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.927+0000 7f627aa00700 1 -- 172.21.15.27:0/304669092 learned_addr learned my addr 172.21.15.27:0/304669092 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:46:36.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.927+0000 7f627a000700 1 --2- 172.21.15.27:0/304669092 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f627c1046d0 0x7f627c1144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:36.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.927+0000 7f627aa00700 1 -- 172.21.15.27:0/304669092 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f627c1046d0 msgr2=0x7f627c1144a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:36.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.927+0000 7f627aa00700 1 --2- 172.21.15.27:0/304669092 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f627c1046d0 0x7f627c1144a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:36.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.927+0000 7f627aa00700 1 -- 172.21.15.27:0/304669092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f626c005040 con 0x7f627c103420 2024-04-16T05:46:36.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.927+0000 7f627a000700 1 --2- 172.21.15.27:0/304669092 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f627c1046d0 0x7f627c1144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:46:36.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.928+0000 7f627aa00700 1 --2- 172.21.15.27:0/304669092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f627c103420 0x7f627c113f60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f626c00b3b0 tx=0x7f626c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:36.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.928+0000 7f6272a00700 1 -- 172.21.15.27:0/304669092 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f626c00e040 con 0x7f627c103420 2024-04-16T05:46:36.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.928+0000 7f6272a00700 1 -- 172.21.15.27:0/304669092 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f626c00bcc0 con 0x7f627c103420 2024-04-16T05:46:36.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.928+0000 7f6281e00700 1 -- 172.21.15.27:0/304669092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f627c119610 con 0x7f627c103420 2024-04-16T05:46:36.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.928+0000 7f6272a00700 1 -- 172.21.15.27:0/304669092 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f626c01c630 con 0x7f627c103420 2024-04-16T05:46:36.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.928+0000 7f6281e00700 1 -- 172.21.15.27:0/304669092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f627c119bd0 con 0x7f627c103420 2024-04-16T05:46:36.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.929+0000 7f6281e00700 1 -- 172.21.15.27:0/304669092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f627c04c710 con 0x7f627c103420 2024-04-16T05:46:36.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.931+0000 7f6272a00700 1 -- 172.21.15.27:0/304669092 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f626c013980 con 0x7f627c103420 2024-04-16T05:46:36.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.932+0000 7f6272a00700 1 --2- 172.21.15.27:0/304669092 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f626006ed30 0x7f62600711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:36.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.932+0000 7f6272a00700 1 -- 172.21.15.27:0/304669092 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f626c08f7f0 con 0x7f627c103420 2024-04-16T05:46:36.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.932+0000 7f627a000700 1 --2- 172.21.15.27:0/304669092 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f626006ed30 0x7f62600711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:36.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.933+0000 7f627a000700 1 --2- 172.21.15.27:0/304669092 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f626006ed30 0x7f62600711e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6268007fd0 tx=0x7f6268009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:36.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:36.937+0000 7f6272a00700 1 -- 172.21.15.27:0/304669092 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f626c05beb0 con 0x7f627c103420 2024-04-16T05:46:37.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.180+0000 7f6281e00700 1 -- 172.21.15.27:0/304669092 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f627c108a60 con 0x7f626006ed30 2024-04-16T05:46:37.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.210+0000 7f6272a00700 1 -- 172.21.15.27:0/304669092 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f627c108a60 con 0x7f626006ed30 2024-04-16T05:46:37.209 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:46:37.209 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (2m) 60s ago 13m 13.5M - 0.25.0 c8568f914cd2 4ed63a8a6b33 2024-04-16T05:46:37.209 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (13m) 60s ago 13m 6665k - 17.2.7-847-g45317f95 b333581ccdf5 f1acb6959c63 2024-04-16T05:46:37.209 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (12m) 39s ago 12m 6660k - 17.2.7-847-g45317f95 b333581ccdf5 e81efc20e8c0 2024-04-16T05:46:37.209 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (2m) 60s ago 13m 75.3M - 9.4.7 954c08fa6188 f7ee2a1afd12 2024-04-16T05:46:37.209 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (6m) 60s ago 6m 23.4M - 17.2.7-847-g45317f95 b333581ccdf5 0824e2621be5 2024-04-16T05:46:37.210 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (6m) 60s ago 6m 13.1M - 17.2.7-847-g45317f95 b333581ccdf5 b332a72cc594 2024-04-16T05:46:37.210 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (6m) 39s ago 6m 13.3M - 17.2.7-847-g45317f95 b333581ccdf5 f5e36815ddfa 2024-04-16T05:46:37.210 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (6m) 39s ago 6m 13.2M - 17.2.7-847-g45317f95 b333581ccdf5 755a73c65a7f 2024-04-16T05:46:37.210 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:8443,9283,8765 running (2m) 60s ago 15m 478M - 19.0.0-2884-g218a6914 a4e4811815e4 b69a0b08e12b 2024-04-16T05:46:37.210 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283,8765 running (99s) 39s ago 12m 451M - 19.0.0-2884-g218a6914 a4e4811815e4 32e27bcd7e42 2024-04-16T05:46:37.210 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (78s) 60s ago 15m 29.4M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 299ca337e4ca 2024-04-16T05:46:37.210 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (45s) 39s ago 12m 21.5M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5db5f3037ed0 2024-04-16T05:46:37.210 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (3m) 60s ago 13m 7696k - 1.5.0 0da6a335fe13 5e29e29868b0 2024-04-16T05:46:37.210 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (2m) 39s ago 12m 7646k - 1.5.0 0da6a335fe13 a360922bc621 2024-04-16T05:46:37.210 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (11m) 60s ago 11m 47.7M 1557M 17.2.7-847-g45317f95 b333581ccdf5 15e049573bca 2024-04-16T05:46:37.210 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (10m) 60s ago 10m 47.3M 1557M 17.2.7-847-g45317f95 b333581ccdf5 e146295f6500 2024-04-16T05:46:37.211 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (9m) 60s ago 9m 49.0M 1557M 17.2.7-847-g45317f95 b333581ccdf5 6212f6a18fa4 2024-04-16T05:46:37.211 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (9m) 39s ago 9m 45.8M 2581M 17.2.7-847-g45317f95 b333581ccdf5 d6724ad05f9f 2024-04-16T05:46:37.211 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (8m) 39s ago 8m 44.6M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7a59908a3e10 2024-04-16T05:46:37.211 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (8m) 39s ago 8m 43.9M 2581M 17.2.7-847-g45317f95 b333581ccdf5 8e4b60163907 2024-04-16T05:46:37.211 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (107s) 60s ago 12m 44.4M - 2.43.0 a07b618ecd1d e70f55ba5ec0 2024-04-16T05:46:37.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.213+0000 7f6281e00700 1 -- 172.21.15.27:0/304669092 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f626006ed30 msgr2=0x7f62600711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:37.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.213+0000 7f6281e00700 1 --2- 172.21.15.27:0/304669092 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f626006ed30 0x7f62600711e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6268007fd0 tx=0x7f6268009040 comp rx=0 tx=0).stop 2024-04-16T05:46:37.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.213+0000 7f6281e00700 1 -- 172.21.15.27:0/304669092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f627c103420 msgr2=0x7f627c113f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:37.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.213+0000 7f6281e00700 1 --2- 172.21.15.27:0/304669092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f627c103420 0x7f627c113f60 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f626c00b3b0 tx=0x7f626c009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:37.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.213+0000 7f6281e00700 1 -- 172.21.15.27:0/304669092 shutdown_connections 2024-04-16T05:46:37.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.213+0000 7f6281e00700 1 --2- 172.21.15.27:0/304669092 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f626006ed30 0x7f62600711e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:37.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.213+0000 7f6281e00700 1 --2- 172.21.15.27:0/304669092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f627c103420 0x7f627c113f60 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:37.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.213+0000 7f6281e00700 1 --2- 172.21.15.27:0/304669092 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f627c1046d0 0x7f627c1144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:37.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.213+0000 7f6281e00700 1 -- 172.21.15.27:0/304669092 >> 172.21.15.27:0/304669092 conn(0x7f627c072640 msgr2=0x7f627c073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:37.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.214+0000 7f6281e00700 1 -- 172.21.15.27:0/304669092 shutdown_connections 2024-04-16T05:46:37.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.214+0000 7f6281e00700 1 -- 172.21.15.27:0/304669092 wait complete. 2024-04-16T05:46:37.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.370+0000 7f5807200700 1 -- 172.21.15.27:0/370900712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5808104520 msgr2=0x7f5808104990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:37.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.370+0000 7f5807200700 1 --2- 172.21.15.27:0/370900712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5808104520 0x7f5808104990 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f57f4004940 tx=0x7f57f4009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:37.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.371+0000 7f5807200700 1 -- 172.21.15.27:0/370900712 shutdown_connections 2024-04-16T05:46:37.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.371+0000 7f5807200700 1 --2- 172.21.15.27:0/370900712 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5808104520 0x7f5808104990 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:37.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.371+0000 7f5807200700 1 --2- 172.21.15.27:0/370900712 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5808103270 0x7f5808103680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:37.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.371+0000 7f5807200700 1 -- 172.21.15.27:0/370900712 >> 172.21.15.27:0/370900712 conn(0x7f5808072530 msgr2=0x7f5808074940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:37.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.371+0000 7f5807200700 1 -- 172.21.15.27:0/370900712 shutdown_connections 2024-04-16T05:46:37.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.371+0000 7f5807200700 1 -- 172.21.15.27:0/370900712 wait complete. 2024-04-16T05:46:37.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.372+0000 7f5807200700 1 Processor -- start 2024-04-16T05:46:37.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.372+0000 7f5807200700 1 -- start start 2024-04-16T05:46:37.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.372+0000 7f5807200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5808103270 0x7f5808113e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:37.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.372+0000 7f5807200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5808104520 0x7f58081143a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:37.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.372+0000 7f5807200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5808115110 con 0x7f5808104520 2024-04-16T05:46:37.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.372+0000 7f5807200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5808115280 con 0x7f5808103270 2024-04-16T05:46:37.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.372+0000 7f5805400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5808104520 0x7f58081143a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:37.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.372+0000 7f5805400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5808104520 0x7f58081143a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34000/0 (socket says 172.21.15.27:34000) 2024-04-16T05:46:37.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.372+0000 7f5805400700 1 -- 172.21.15.27:0/488993366 learned_addr learned my addr 172.21.15.27:0/488993366 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:46:37.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.373+0000 7f5805e00700 1 --2- 172.21.15.27:0/488993366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5808103270 0x7f5808113e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:37.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.373+0000 7f5805400700 1 -- 172.21.15.27:0/488993366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5808103270 msgr2=0x7f5808113e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:37.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.373+0000 7f5805400700 1 --2- 172.21.15.27:0/488993366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5808103270 0x7f5808113e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:37.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.373+0000 7f5805400700 1 -- 172.21.15.27:0/488993366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f4005040 con 0x7f5808104520 2024-04-16T05:46:37.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.373+0000 7f5805e00700 1 --2- 172.21.15.27:0/488993366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5808103270 0x7f5808113e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:46:37.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.373+0000 7f5805400700 1 --2- 172.21.15.27:0/488993366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5808104520 0x7f58081143a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f57f400b0d0 tx=0x7f57f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:37.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.373+0000 7f57fe000700 1 -- 172.21.15.27:0/488993366 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57f400e070 con 0x7f5808104520 2024-04-16T05:46:37.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.373+0000 7f57fe000700 1 -- 172.21.15.27:0/488993366 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57f400b410 con 0x7f5808104520 2024-04-16T05:46:37.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.373+0000 7f57fe000700 1 -- 172.21.15.27:0/488993366 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57f4012600 con 0x7f5808104520 2024-04-16T05:46:37.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.373+0000 7f5807200700 1 -- 172.21.15.27:0/488993366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58081154a0 con 0x7f5808104520 2024-04-16T05:46:37.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.374+0000 7f5807200700 1 -- 172.21.15.27:0/488993366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5808115a60 con 0x7f5808104520 2024-04-16T05:46:37.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.375+0000 7f5807200700 1 -- 172.21.15.27:0/488993366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58080760a0 con 0x7f5808104520 2024-04-16T05:46:37.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.376+0000 7f57fe000700 1 -- 172.21.15.27:0/488993366 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f57f400ba80 con 0x7f5808104520 2024-04-16T05:46:37.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.377+0000 7f57fe000700 1 --2- 172.21.15.27:0/488993366 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f57ec06ed30 0x7f57ec0711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:37.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.377+0000 7f57fe000700 1 -- 172.21.15.27:0/488993366 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f57f408f800 con 0x7f5808104520 2024-04-16T05:46:37.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.377+0000 7f5805e00700 1 --2- 172.21.15.27:0/488993366 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f57ec06ed30 0x7f57ec0711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:37.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.378+0000 7f5805e00700 1 --2- 172.21.15.27:0/488993366 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f57ec06ed30 0x7f57ec0711e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f57f8007fd0 tx=0x7f57f800b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:37.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.382+0000 7f57fe000700 1 -- 172.21.15.27:0/488993366 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f57f4058440 con 0x7f5808104520 2024-04-16T05:46:37.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:37 smithi027 ceph-mon[93635]: from='client.34184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:46:37.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.704+0000 7f5807200700 1 -- 172.21.15.27:0/488993366 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f580804bed0 con 0x7f5808104520 2024-04-16T05:46:37.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.705+0000 7f57fe000700 1 -- 172.21.15.27:0/488993366 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+736 (secure 0 0 0) 0x7f57f405bb00 con 0x7f5808104520 2024-04-16T05:46:37.704 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:46:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:46:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:46:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:46:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:46:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:46:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:46:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:46:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T05:46:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:46:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:46:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T05:46:37.704 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:46:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:46:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 10, 2024-04-16T05:46:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T05:46:37.705 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:46:37.705 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:46:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.709+0000 7f5807200700 1 -- 172.21.15.27:0/488993366 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f57ec06ed30 msgr2=0x7f57ec0711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.709+0000 7f5807200700 1 --2- 172.21.15.27:0/488993366 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f57ec06ed30 0x7f57ec0711e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f57f8007fd0 tx=0x7f57f800b040 comp rx=0 tx=0).stop 2024-04-16T05:46:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.709+0000 7f5807200700 1 -- 172.21.15.27:0/488993366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5808104520 msgr2=0x7f58081143a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.709+0000 7f5807200700 1 --2- 172.21.15.27:0/488993366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5808104520 0x7f58081143a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f57f400b0d0 tx=0x7f57f4009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.709+0000 7f5807200700 1 -- 172.21.15.27:0/488993366 shutdown_connections 2024-04-16T05:46:37.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.709+0000 7f5807200700 1 --2- 172.21.15.27:0/488993366 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f57ec06ed30 0x7f57ec0711e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:37.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.709+0000 7f5807200700 1 --2- 172.21.15.27:0/488993366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5808103270 0x7f5808113e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:37.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.709+0000 7f5807200700 1 --2- 172.21.15.27:0/488993366 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5808104520 0x7f58081143a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:37.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.709+0000 7f5807200700 1 -- 172.21.15.27:0/488993366 >> 172.21.15.27:0/488993366 conn(0x7f5808072530 msgr2=0x7f5808074b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:37.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.709+0000 7f5807200700 1 -- 172.21.15.27:0/488993366 shutdown_connections 2024-04-16T05:46:37.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.709+0000 7f5807200700 1 -- 172.21.15.27:0/488993366 wait complete. 2024-04-16T05:46:37.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:37 smithi122 ceph-mon[80930]: from='client.34184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:46:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.872+0000 7f3a9e000700 1 -- 172.21.15.27:0/1616994517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a981046d0 msgr2=0x7f3a98104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.872+0000 7f3a9e000700 1 --2- 172.21.15.27:0/1616994517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a981046d0 0x7f3a98104b40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3a84004940 tx=0x7f3a84009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.872+0000 7f3a9e000700 1 -- 172.21.15.27:0/1616994517 shutdown_connections 2024-04-16T05:46:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.872+0000 7f3a9e000700 1 --2- 172.21.15.27:0/1616994517 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a981046d0 0x7f3a98104b40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.872+0000 7f3a9e000700 1 --2- 172.21.15.27:0/1616994517 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a98103420 0x7f3a98103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.872+0000 7f3a9e000700 1 -- 172.21.15.27:0/1616994517 >> 172.21.15.27:0/1616994517 conn(0x7f3a98072640 msgr2=0x7f3a98074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.872+0000 7f3a9e000700 1 -- 172.21.15.27:0/1616994517 shutdown_connections 2024-04-16T05:46:37.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.872+0000 7f3a9e000700 1 -- 172.21.15.27:0/1616994517 wait complete. 2024-04-16T05:46:37.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.873+0000 7f3a9e000700 1 Processor -- start 2024-04-16T05:46:37.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.873+0000 7f3a9e000700 1 -- start start 2024-04-16T05:46:37.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.874+0000 7f3a9e000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a98103420 0x7f3a98113ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:37.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.874+0000 7f3a9e000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a981046d0 0x7f3a98114430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:37.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.874+0000 7f3a9e000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a98114a30 con 0x7f3a98103420 2024-04-16T05:46:37.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.874+0000 7f3a9e000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a98114ba0 con 0x7f3a981046d0 2024-04-16T05:46:37.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.874+0000 7f3a97400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a98103420 0x7f3a98113ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:37.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.874+0000 7f3a97400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a98103420 0x7f3a98113ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34014/0 (socket says 172.21.15.27:34014) 2024-04-16T05:46:37.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.874+0000 7f3a97400700 1 -- 172.21.15.27:0/916787885 learned_addr learned my addr 172.21.15.27:0/916787885 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:46:37.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.875+0000 7f3a96a00700 1 --2- 172.21.15.27:0/916787885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a981046d0 0x7f3a98114430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:37.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.875+0000 7f3a97400700 1 -- 172.21.15.27:0/916787885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a981046d0 msgr2=0x7f3a98114430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:37.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.875+0000 7f3a97400700 1 --2- 172.21.15.27:0/916787885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a981046d0 0x7f3a98114430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:37.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.875+0000 7f3a97400700 1 -- 172.21.15.27:0/916787885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a84005040 con 0x7f3a98103420 2024-04-16T05:46:37.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.875+0000 7f3a97400700 1 --2- 172.21.15.27:0/916787885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a98103420 0x7f3a98113ef0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3a8800eb70 tx=0x7f3a8800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:37.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.876+0000 7f3a8f400700 1 -- 172.21.15.27:0/916787885 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a88007cc0 con 0x7f3a98103420 2024-04-16T05:46:37.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.876+0000 7f3a8f400700 1 -- 172.21.15.27:0/916787885 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a8800f040 con 0x7f3a98103420 2024-04-16T05:46:37.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.876+0000 7f3a8f400700 1 -- 172.21.15.27:0/916787885 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a8800ca90 con 0x7f3a98103420 2024-04-16T05:46:37.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.876+0000 7f3a9e000700 1 -- 172.21.15.27:0/916787885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a981195a0 con 0x7f3a98103420 2024-04-16T05:46:37.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.876+0000 7f3a9e000700 1 -- 172.21.15.27:0/916787885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a98119b60 con 0x7f3a98103420 2024-04-16T05:46:37.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.877+0000 7f3a9e000700 1 -- 172.21.15.27:0/916787885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a9804c710 con 0x7f3a98103420 2024-04-16T05:46:37.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.878+0000 7f3a8f400700 1 -- 172.21.15.27:0/916787885 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f3a8800cbf0 con 0x7f3a98103420 2024-04-16T05:46:37.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.879+0000 7f3a8f400700 1 --2- 172.21.15.27:0/916787885 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3a7c06ed30 0x7f3a7c0711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:37.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.879+0000 7f3a8f400700 1 -- 172.21.15.27:0/916787885 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f3a8808e5b0 con 0x7f3a98103420 2024-04-16T05:46:37.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.880+0000 7f3a96a00700 1 --2- 172.21.15.27:0/916787885 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3a7c06ed30 0x7f3a7c0711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:37.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.881+0000 7f3a96a00700 1 --2- 172.21.15.27:0/916787885 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3a7c06ed30 0x7f3a7c0711e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3a84009fd0 tx=0x7f3a84009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:37.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:37.884+0000 7f3a8f400700 1 -- 172.21.15.27:0/916787885 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3a8805acf0 con 0x7f3a98103420 2024-04-16T05:46:38.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.161+0000 7f3a9e000700 1 -- 172.21.15.27:0/916787885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f3a98119fb0 con 0x7f3a98103420 2024-04-16T05:46:38.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.162+0000 7f3a8f400700 1 -- 172.21.15.27:0/916787885 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1880 (secure 0 0 0) 0x7f3a8805a880 con 0x7f3a98103420 2024-04-16T05:46:38.161 INFO:teuthology.orchestra.run.smithi027.stdout:e16 2024-04-16T05:46:38.161 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:46:38.161 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:46:38.161 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:46:38.162 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:46:38.162 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:46:38.162 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:46:38.162 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 15 2024-04-16T05:46:38.162 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:46:38.162 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:46:38.162 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:40:32.701649+0000 2024-04-16T05:46:38.162 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:46:38.162 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:46:38.162 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:46:38.162 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:46:38.162 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:46:38.162 INFO:teuthology.orchestra.run.smithi027.stdout:max_xattr_size 65536 2024-04-16T05:46:38.162 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:46:38.163 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:46:38.163 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 46 2024-04-16T05:46:38.163 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:46:38.163 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:46:38.163 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:46:38.163 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=14462} 2024-04-16T05:46:38.163 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:46:38.163 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:46:38.163 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:46:38.163 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:46:38.163 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:46:38.163 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:46:38.163 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:46:38.163 INFO:teuthology.orchestra.run.smithi027.stdout:bal_rank_mask -1 2024-04-16T05:46:38.163 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:46:38.164 INFO:teuthology.orchestra.run.smithi027.stdout:qdb_cluster leader: 0 members: 2024-04-16T05:46:38.164 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:46:38.164 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:46:38.164 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:46:38.164 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:46:38.164 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:46:38.164 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{-1:14474} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2065690722,v1:172.21.15.27:6827/2065690722] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:46:38.164 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:24263} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:46:38.164 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:24273} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:46:38.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.165+0000 7f3a9e000700 1 -- 172.21.15.27:0/916787885 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3a7c06ed30 msgr2=0x7f3a7c0711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:38.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.165+0000 7f3a9e000700 1 --2- 172.21.15.27:0/916787885 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3a7c06ed30 0x7f3a7c0711e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3a84009fd0 tx=0x7f3a84009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:38.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.165+0000 7f3a9e000700 1 -- 172.21.15.27:0/916787885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a98103420 msgr2=0x7f3a98113ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:38.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.165+0000 7f3a9e000700 1 --2- 172.21.15.27:0/916787885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a98103420 0x7f3a98113ef0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3a8800eb70 tx=0x7f3a8800b040 comp rx=0 tx=0).stop 2024-04-16T05:46:38.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.165+0000 7f3a9e000700 1 -- 172.21.15.27:0/916787885 shutdown_connections 2024-04-16T05:46:38.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.165+0000 7f3a9e000700 1 --2- 172.21.15.27:0/916787885 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3a7c06ed30 0x7f3a7c0711e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:38.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.165+0000 7f3a9e000700 1 --2- 172.21.15.27:0/916787885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a98103420 0x7f3a98113ef0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:38.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.165+0000 7f3a9e000700 1 --2- 172.21.15.27:0/916787885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3a981046d0 0x7f3a98114430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:38.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.165+0000 7f3a9e000700 1 -- 172.21.15.27:0/916787885 >> 172.21.15.27:0/916787885 conn(0x7f3a98072640 msgr2=0x7f3a98073c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:38.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.166+0000 7f3a9e000700 1 -- 172.21.15.27:0/916787885 shutdown_connections 2024-04-16T05:46:38.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.166+0000 7f3a9e000700 1 -- 172.21.15.27:0/916787885 wait complete. 2024-04-16T05:46:38.166 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2024-04-16T05:46:38.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.320+0000 7f7c87e00700 1 -- 172.21.15.27:0/3360593452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c88103270 msgr2=0x7f7c88103680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:38.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.320+0000 7f7c87e00700 1 --2- 172.21.15.27:0/3360593452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c88103270 0x7f7c88103680 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7c78004940 tx=0x7f7c78009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:38.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.320+0000 7f7c87e00700 1 -- 172.21.15.27:0/3360593452 shutdown_connections 2024-04-16T05:46:38.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.320+0000 7f7c87e00700 1 --2- 172.21.15.27:0/3360593452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c88104520 0x7f7c88104990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:38.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.320+0000 7f7c87e00700 1 --2- 172.21.15.27:0/3360593452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c88103270 0x7f7c88103680 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:38.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.320+0000 7f7c87e00700 1 -- 172.21.15.27:0/3360593452 >> 172.21.15.27:0/3360593452 conn(0x7f7c88072530 msgr2=0x7f7c88074940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:38.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.320+0000 7f7c87e00700 1 -- 172.21.15.27:0/3360593452 shutdown_connections 2024-04-16T05:46:38.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.321+0000 7f7c87e00700 1 -- 172.21.15.27:0/3360593452 wait complete. 2024-04-16T05:46:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.321+0000 7f7c87e00700 1 Processor -- start 2024-04-16T05:46:38.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.322+0000 7f7c87e00700 1 -- start start 2024-04-16T05:46:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.322+0000 7f7c87e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c88103270 0x7f7c88113e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.322+0000 7f7c87e00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c88104520 0x7f7c88114360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.322+0000 7f7c87e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c88115100 con 0x7f7c88103270 2024-04-16T05:46:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.322+0000 7f7c87e00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c88115270 con 0x7f7c88104520 2024-04-16T05:46:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.322+0000 7f7c86a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c88103270 0x7f7c88113e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.322+0000 7f7c86a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c88103270 0x7f7c88113e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34022/0 (socket says 172.21.15.27:34022) 2024-04-16T05:46:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.322+0000 7f7c86a00700 1 -- 172.21.15.27:0/1189208352 learned_addr learned my addr 172.21.15.27:0/1189208352 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:46:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.322+0000 7f7c86000700 1 --2- 172.21.15.27:0/1189208352 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c88104520 0x7f7c88114360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.323+0000 7f7c86a00700 1 -- 172.21.15.27:0/1189208352 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c88104520 msgr2=0x7f7c88114360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.323+0000 7f7c86a00700 1 --2- 172.21.15.27:0/1189208352 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c88104520 0x7f7c88114360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.323+0000 7f7c86a00700 1 -- 172.21.15.27:0/1189208352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c78005040 con 0x7f7c88103270 2024-04-16T05:46:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.323+0000 7f7c86a00700 1 --2- 172.21.15.27:0/1189208352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c88103270 0x7f7c88113e20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7c78013870 tx=0x7f7c78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.323+0000 7f7c7ea00700 1 -- 172.21.15.27:0/1189208352 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c7800e070 con 0x7f7c88103270 2024-04-16T05:46:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.323+0000 7f7c7ea00700 1 -- 172.21.15.27:0/1189208352 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c78013de0 con 0x7f7c88103270 2024-04-16T05:46:38.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.323+0000 7f7c87e00700 1 -- 172.21.15.27:0/1189208352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c881154f0 con 0x7f7c88103270 2024-04-16T05:46:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.323+0000 7f7c7ea00700 1 -- 172.21.15.27:0/1189208352 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c7801cea0 con 0x7f7c88103270 2024-04-16T05:46:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.323+0000 7f7c87e00700 1 -- 172.21.15.27:0/1189208352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c881159d0 con 0x7f7c88103270 2024-04-16T05:46:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.324+0000 7f7c87e00700 1 -- 172.21.15.27:0/1189208352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c8810c570 con 0x7f7c88103270 2024-04-16T05:46:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.326+0000 7f7c7ea00700 1 -- 172.21.15.27:0/1189208352 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f7c7801c690 con 0x7f7c88103270 2024-04-16T05:46:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.327+0000 7f7c7ea00700 1 --2- 172.21.15.27:0/1189208352 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7c6c06ed30 0x7f7c6c0711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:38.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.327+0000 7f7c7ea00700 1 -- 172.21.15.27:0/1189208352 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f7c78090290 con 0x7f7c88103270 2024-04-16T05:46:38.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.327+0000 7f7c86000700 1 --2- 172.21.15.27:0/1189208352 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7c6c06ed30 0x7f7c6c0711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:38.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.328+0000 7f7c86000700 1 --2- 172.21.15.27:0/1189208352 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7c6c06ed30 0x7f7c6c0711e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7c88113bb0 tx=0x7f7c7400b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:38.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.331+0000 7f7c7ea00700 1 -- 172.21.15.27:0/1189208352 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7c78058e20 con 0x7f7c88103270 2024-04-16T05:46:38.437 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:38 smithi027 ceph-mon[93635]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:46:38.437 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:38 smithi027 ceph-mon[93635]: pgmap v37: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:38.437 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:38 smithi027 ceph-mon[93635]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:46:38.437 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:38 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:38.437 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:38 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:38.437 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:38 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T05:46:38.437 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:38 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:38.437 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:38 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/488993366' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:46:38.437 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:38 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/916787885' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:46:38.544 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:38 smithi122 ceph-mon[80930]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:46:38.544 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:38 smithi122 ceph-mon[80930]: pgmap v37: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:38.545 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:38 smithi122 ceph-mon[80930]: from='client.34192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:46:38.545 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:38 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:38.545 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:38 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:38.545 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:38 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T05:46:38.545 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:38 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:38.545 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:38 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/488993366' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:46:38.545 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:38 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/916787885' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:46:38.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.575+0000 7f7c87e00700 1 -- 172.21.15.27:0/1189208352 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c88115cb0 con 0x7f7c6c06ed30 2024-04-16T05:46:38.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.578+0000 7f7c7ea00700 1 -- 172.21.15.27:0/1189208352 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+383 (secure 0 0 0) 0x7f7c88115cb0 con 0x7f7c6c06ed30 2024-04-16T05:46:38.577 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:46:38.577 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T05:46:38.577 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:46:38.577 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:46:38.577 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [ 2024-04-16T05:46:38.577 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr", 2024-04-16T05:46:38.577 INFO:teuthology.orchestra.run.smithi027.stdout: "mon" 2024-04-16T05:46:38.578 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-16T05:46:38.578 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "4/21 daemons upgraded", 2024-04-16T05:46:38.578 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Currently upgrading mon daemons", 2024-04-16T05:46:38.578 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:46:38.578 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:46:38.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.582+0000 7f7c87e00700 1 -- 172.21.15.27:0/1189208352 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7c6c06ed30 msgr2=0x7f7c6c0711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:38.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.582+0000 7f7c87e00700 1 --2- 172.21.15.27:0/1189208352 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7c6c06ed30 0x7f7c6c0711e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7c88113bb0 tx=0x7f7c7400b040 comp rx=0 tx=0).stop 2024-04-16T05:46:38.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.582+0000 7f7c87e00700 1 -- 172.21.15.27:0/1189208352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c88103270 msgr2=0x7f7c88113e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:38.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.582+0000 7f7c87e00700 1 --2- 172.21.15.27:0/1189208352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c88103270 0x7f7c88113e20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7c78013870 tx=0x7f7c78009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:38.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.582+0000 7f7c87e00700 1 -- 172.21.15.27:0/1189208352 shutdown_connections 2024-04-16T05:46:38.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.582+0000 7f7c87e00700 1 --2- 172.21.15.27:0/1189208352 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7c6c06ed30 0x7f7c6c0711e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:38.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.582+0000 7f7c87e00700 1 --2- 172.21.15.27:0/1189208352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c88103270 0x7f7c88113e20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:38.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.582+0000 7f7c87e00700 1 --2- 172.21.15.27:0/1189208352 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7c88104520 0x7f7c88114360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:38.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.582+0000 7f7c87e00700 1 -- 172.21.15.27:0/1189208352 >> 172.21.15.27:0/1189208352 conn(0x7f7c88072530 msgr2=0x7f7c88074b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:38.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.582+0000 7f7c87e00700 1 -- 172.21.15.27:0/1189208352 shutdown_connections 2024-04-16T05:46:38.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.582+0000 7f7c87e00700 1 -- 172.21.15.27:0/1189208352 wait complete. 2024-04-16T05:46:38.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.741+0000 7f82c8400700 1 -- 172.21.15.27:0/2614217613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82c01046d0 msgr2=0x7f82c0104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:38.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.741+0000 7f82c8400700 1 --2- 172.21.15.27:0/2614217613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82c01046d0 0x7f82c0104b40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f82b0004940 tx=0x7f82b0009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:38.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.741+0000 7f82c8400700 1 -- 172.21.15.27:0/2614217613 shutdown_connections 2024-04-16T05:46:38.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.741+0000 7f82c8400700 1 --2- 172.21.15.27:0/2614217613 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82c01046d0 0x7f82c0104b40 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:38.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.741+0000 7f82c8400700 1 --2- 172.21.15.27:0/2614217613 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82c0103420 0x7f82c0103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:38.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.741+0000 7f82c8400700 1 -- 172.21.15.27:0/2614217613 >> 172.21.15.27:0/2614217613 conn(0x7f82c0072640 msgr2=0x7f82c0074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:38.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.741+0000 7f82c8400700 1 -- 172.21.15.27:0/2614217613 shutdown_connections 2024-04-16T05:46:38.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.741+0000 7f82c8400700 1 -- 172.21.15.27:0/2614217613 wait complete. 2024-04-16T05:46:38.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.742+0000 7f82c8400700 1 Processor -- start 2024-04-16T05:46:38.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.742+0000 7f82c8400700 1 -- start start 2024-04-16T05:46:38.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.742+0000 7f82c8400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82c0103420 0x7f82c0113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:38.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.742+0000 7f82c8400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82c01046d0 0x7f82c0114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:38.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.742+0000 7f82c8400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c0114b20 con 0x7f82c0103420 2024-04-16T05:46:38.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.742+0000 7f82c8400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c01194c0 con 0x7f82c01046d0 2024-04-16T05:46:38.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.743+0000 7f82c5a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82c0103420 0x7f82c0113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:38.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.743+0000 7f82c5a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82c0103420 0x7f82c0113f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34030/0 (socket says 172.21.15.27:34030) 2024-04-16T05:46:38.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.743+0000 7f82c5a00700 1 -- 172.21.15.27:0/1436903921 learned_addr learned my addr 172.21.15.27:0/1436903921 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:46:38.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.743+0000 7f82c5000700 1 --2- 172.21.15.27:0/1436903921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82c01046d0 0x7f82c0114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:38.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.743+0000 7f82c5a00700 1 -- 172.21.15.27:0/1436903921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82c01046d0 msgr2=0x7f82c0114490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.743+0000 7f82c5a00700 1 --2- 172.21.15.27:0/1436903921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82c01046d0 0x7f82c0114490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.743+0000 7f82c5a00700 1 -- 172.21.15.27:0/1436903921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b0005040 con 0x7f82c0103420 2024-04-16T05:46:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.744+0000 7f82c5a00700 1 --2- 172.21.15.27:0/1436903921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82c0103420 0x7f82c0113f50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f82b400b790 tx=0x7f82b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.744+0000 7f82bd600700 1 -- 172.21.15.27:0/1436903921 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82b400f770 con 0x7f82c0103420 2024-04-16T05:46:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.744+0000 7f82bd600700 1 -- 172.21.15.27:0/1436903921 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82b400fda0 con 0x7f82c0103420 2024-04-16T05:46:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.744+0000 7f82bd600700 1 -- 172.21.15.27:0/1436903921 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82b400d310 con 0x7f82c0103420 2024-04-16T05:46:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.744+0000 7f82c8400700 1 -- 172.21.15.27:0/1436903921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c0119720 con 0x7f82c0103420 2024-04-16T05:46:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.744+0000 7f82c8400700 1 -- 172.21.15.27:0/1436903921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c0119c90 con 0x7f82c0103420 2024-04-16T05:46:38.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.745+0000 7f82c8400700 1 -- 172.21.15.27:0/1436903921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c004c710 con 0x7f82c0103420 2024-04-16T05:46:38.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.747+0000 7f82bd600700 1 -- 172.21.15.27:0/1436903921 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 38) v1 ==== 92847+0+0 (secure 0 0 0) 0x7f82b4010420 con 0x7f82c0103420 2024-04-16T05:46:38.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.748+0000 7f82bd600700 1 --2- 172.21.15.27:0/1436903921 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f82a806ed30 0x7f82a80711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:46:38.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.748+0000 7f82bd600700 1 -- 172.21.15.27:0/1436903921 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f82b408f130 con 0x7f82c0103420 2024-04-16T05:46:38.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.749+0000 7f82c5000700 1 --2- 172.21.15.27:0/1436903921 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f82a806ed30 0x7f82a80711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:46:38.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.749+0000 7f82c5000700 1 --2- 172.21.15.27:0/1436903921 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f82a806ed30 0x7f82a80711e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f82b0004f90 tx=0x7f82b0009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:46:38.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:38.752+0000 7f82bd600700 1 -- 172.21.15.27:0/1436903921 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f82b405b980 con 0x7f82c0103420 2024-04-16T05:46:39.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:39.072+0000 7f82c8400700 1 -- 172.21.15.27:0/1436903921 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f82c010bf90 con 0x7f82c0103420 2024-04-16T05:46:39.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:39.072+0000 7f82bd600700 1 -- 172.21.15.27:0/1436903921 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f82b405b510 con 0x7f82c0103420 2024-04-16T05:46:39.071 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:46:39.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:39.075+0000 7f82c8400700 1 -- 172.21.15.27:0/1436903921 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f82a806ed30 msgr2=0x7f82a80711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:39.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:39.075+0000 7f82c8400700 1 --2- 172.21.15.27:0/1436903921 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f82a806ed30 0x7f82a80711e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f82b0004f90 tx=0x7f82b0009e30 comp rx=0 tx=0).stop 2024-04-16T05:46:39.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:39.075+0000 7f82c8400700 1 -- 172.21.15.27:0/1436903921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82c0103420 msgr2=0x7f82c0113f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:46:39.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:39.075+0000 7f82c8400700 1 --2- 172.21.15.27:0/1436903921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82c0103420 0x7f82c0113f50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f82b400b790 tx=0x7f82b4009250 comp rx=0 tx=0).stop 2024-04-16T05:46:39.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:39.075+0000 7f82c8400700 1 -- 172.21.15.27:0/1436903921 shutdown_connections 2024-04-16T05:46:39.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:39.075+0000 7f82c8400700 1 --2- 172.21.15.27:0/1436903921 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f82a806ed30 0x7f82a80711e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:39.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:39.075+0000 7f82c8400700 1 --2- 172.21.15.27:0/1436903921 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f82c0103420 0x7f82c0113f50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:39.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:39.075+0000 7f82c8400700 1 --2- 172.21.15.27:0/1436903921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82c01046d0 0x7f82c0114490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:46:39.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:39.075+0000 7f82c8400700 1 -- 172.21.15.27:0/1436903921 >> 172.21.15.27:0/1436903921 conn(0x7f82c0072640 msgr2=0x7f82c0073ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:46:39.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:39.076+0000 7f82c8400700 1 -- 172.21.15.27:0/1436903921 shutdown_connections 2024-04-16T05:46:39.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:46:39.076+0000 7f82c8400700 1 -- 172.21.15.27:0/1436903921 wait complete. 2024-04-16T05:46:39.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:39 smithi027 ceph-mon[93635]: Reconfiguring osd.5 (monmap changed)... 2024-04-16T05:46:39.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:39 smithi027 ceph-mon[93635]: Reconfiguring daemon osd.5 on smithi122 2024-04-16T05:46:39.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:39 smithi027 ceph-mon[93635]: from='client.34204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:46:39.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:39 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1436903921' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:46:39.559 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:39 smithi122 ceph-mon[80930]: Reconfiguring osd.5 (monmap changed)... 2024-04-16T05:46:39.560 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:39 smithi122 ceph-mon[80930]: Reconfiguring daemon osd.5 on smithi122 2024-04-16T05:46:39.560 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:39 smithi122 ceph-mon[80930]: from='client.34204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:46:39.560 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:39 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1436903921' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:46:40.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:40 smithi027 ceph-mon[93635]: pgmap v38: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:40.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:40.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:40.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.lojjdp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:46:40.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:40.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:40 smithi122 ceph-mon[80930]: pgmap v38: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:40.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:40 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:40.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:40 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:40.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:40 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.lojjdp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:46:40.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:40 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:41.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:41 smithi122 ceph-mon[80930]: Reconfiguring mds.cephfs.smithi122.lojjdp (monmap changed)... 2024-04-16T05:46:41.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:41 smithi122 ceph-mon[80930]: Reconfiguring daemon mds.cephfs.smithi122.lojjdp on smithi122 2024-04-16T05:46:41.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:41 smithi027 ceph-mon[93635]: Reconfiguring mds.cephfs.smithi122.lojjdp (monmap changed)... 2024-04-16T05:46:41.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:41 smithi027 ceph-mon[93635]: Reconfiguring daemon mds.cephfs.smithi122.lojjdp on smithi122 2024-04-16T05:46:42.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:42 smithi122 ceph-mon[80930]: pgmap v39: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:42.708 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:42 smithi027 ceph-mon[93635]: pgmap v39: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:43.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:43 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:43.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:43 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:43.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:43 smithi122 ceph-mon[80930]: Reconfiguring mds.cephfs.smithi122.rulewi (monmap changed)... 2024-04-16T05:46:43.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:43 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.rulewi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:46:43.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:43 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:43.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:43 smithi122 ceph-mon[80930]: Reconfiguring daemon mds.cephfs.smithi122.rulewi on smithi122 2024-04-16T05:46:43.738 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:43 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:43.738 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:43 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:43.738 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:43 smithi027 ceph-mon[93635]: Reconfiguring mds.cephfs.smithi122.rulewi (monmap changed)... 2024-04-16T05:46:43.738 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:43 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.rulewi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:46:43.739 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:43 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:43.739 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:43 smithi027 ceph-mon[93635]: Reconfiguring daemon mds.cephfs.smithi122.rulewi on smithi122 2024-04-16T05:46:44.446 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:44 smithi027 ceph-mon[93635]: pgmap v40: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:44.447 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:46:44.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:44 smithi122 ceph-mon[80930]: pgmap v40: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:44.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:44 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:46:45.710 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:45 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:45.711 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:45 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:45.711 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:45 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:46:45.711 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:45 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:46:45.711 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:45 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:46:45.711 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:45 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:45.711 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:45 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi027"}]: dispatch 2024-04-16T05:46:45.711 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:45 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi027"}]': finished 2024-04-16T05:46:45.711 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:45 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi122"}]: dispatch 2024-04-16T05:46:45.711 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:45 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi122"}]': finished 2024-04-16T05:46:46.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:45 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:46.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:45 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:46.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:45 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:46:46.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:45 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:46:46.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:45 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:46:46.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:45 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:46.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:45 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi027"}]: dispatch 2024-04-16T05:46:46.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:45 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi027"}]': finished 2024-04-16T05:46:46.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:45 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi122"}]: dispatch 2024-04-16T05:46:46.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:45 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi122"}]': finished 2024-04-16T05:46:47.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:46 smithi122 ceph-mon[80930]: Upgrade: Setting container_image for all mon 2024-04-16T05:46:47.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:46 smithi122 ceph-mon[80930]: pgmap v41: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:47.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:46 smithi027 ceph-mon[93635]: Upgrade: Setting container_image for all mon 2024-04-16T05:46:47.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:46 smithi027 ceph-mon[93635]: pgmap v41: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:48.543 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:48 smithi027 ceph-mon[93635]: pgmap v42: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:48.543 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:48 smithi027 ceph-mon[93635]: Upgrade: Updating crash.smithi027 (1/2) 2024-04-16T05:46:48.543 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:48 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:48.543 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:48 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi027", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T05:46:48.543 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:48 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:48.543 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:48 smithi027 ceph-mon[93635]: Deploying daemon crash.smithi027 on smithi027 2024-04-16T05:46:48.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:48 smithi122 ceph-mon[80930]: pgmap v42: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:48.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:48 smithi122 ceph-mon[80930]: Upgrade: Updating crash.smithi027 (1/2) 2024-04-16T05:46:48.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:48 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:48.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:48 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi027", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T05:46:48.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:48 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:48.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:48 smithi122 ceph-mon[80930]: Deploying daemon crash.smithi027 on smithi027 2024-04-16T05:46:50.550 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:50 smithi027 ceph-mon[93635]: pgmap v43: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:50.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:50 smithi122 ceph-mon[80930]: pgmap v43: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:52.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:52 smithi027 ceph-mon[93635]: pgmap v44: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:52.574 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:52 smithi122 ceph-mon[80930]: pgmap v44: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:54.493 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:54 smithi122 ceph-mon[80930]: pgmap v45: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:54.493 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:54 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:54.493 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:54 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:54.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:54 smithi027 ceph-mon[93635]: pgmap v45: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:54.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:54 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:54.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:54 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:56.668 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:56 smithi122 ceph-mon[80930]: pgmap v46: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:56.668 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:56 smithi122 ceph-mon[80930]: Upgrade: Updating crash.smithi122 (2/2) 2024-04-16T05:46:56.668 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:56 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:56.668 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:56 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T05:46:56.668 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:56 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:56.668 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:56 smithi122 ceph-mon[80930]: Deploying daemon crash.smithi122 on smithi122 2024-04-16T05:46:56.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:56 smithi027 ceph-mon[93635]: pgmap v46: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:56.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:56 smithi027 ceph-mon[93635]: Upgrade: Updating crash.smithi122 (2/2) 2024-04-16T05:46:56.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:56 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:46:56.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:56 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi122", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-16T05:46:56.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:56 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:46:56.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:56 smithi027 ceph-mon[93635]: Deploying daemon crash.smithi122 on smithi122 2024-04-16T05:46:58.703 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:58 smithi122 ceph-mon[80930]: pgmap v47: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:58.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:58 smithi027 ceph-mon[93635]: pgmap v47: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:46:59.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:46:59 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:46:59.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:46:59 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:47:00.645 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:00 smithi122 ceph-mon[80930]: pgmap v48: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:00.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:00 smithi027 ceph-mon[93635]: pgmap v48: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:02.209 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:01 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:02.209 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:01 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:02.209 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:01 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:47:02.209 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:01 smithi122 ceph-mon[80930]: pgmap v49: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T05:47:02.290 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:01 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:02.291 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:01 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:02.291 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:01 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:47:02.291 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:01 smithi027 ceph-mon[93635]: pgmap v49: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T05:47:03.131 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:02 smithi122 ceph-mon[80930]: mgrmap e39: smithi027.qgzypi(active, since 93s), standbys: smithi122.sgokkq 2024-04-16T05:47:03.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:02 smithi027 ceph-mon[93635]: mgrmap e39: smithi027.qgzypi(active, since 93s), standbys: smithi122.sgokkq 2024-04-16T05:47:04.268 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:03 smithi122 ceph-mon[80930]: pgmap v50: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T05:47:04.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:03 smithi027 ceph-mon[93635]: pgmap v50: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T05:47:06.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:06 smithi122 ceph-mon[80930]: pgmap v51: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T05:47:06.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:06 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:06.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:06 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:06.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:06 smithi027 ceph-mon[93635]: pgmap v51: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T05:47:06.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:06 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:06.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:06 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:08.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:08 smithi027 ceph-mon[93635]: pgmap v52: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T05:47:08.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:08 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:08.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:08 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:08.593 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:08 smithi122 ceph-mon[80930]: pgmap v52: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T05:47:08.593 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:08 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:08.593 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:08 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.242+0000 7f9434200700 1 -- 172.21.15.27:0/1396663988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f942c0fd710 msgr2=0x7f942c0fe070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.242+0000 7f9434200700 1 --2- 172.21.15.27:0/1396663988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f942c0fd710 0x7f942c0fe070 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9418004940 tx=0x7f9418009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.242+0000 7f9434200700 1 -- 172.21.15.27:0/1396663988 shutdown_connections 2024-04-16T05:47:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.242+0000 7f9434200700 1 --2- 172.21.15.27:0/1396663988 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f942c0fd710 0x7f942c0fe070 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.242+0000 7f9434200700 1 --2- 172.21.15.27:0/1396663988 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f942c0fcdc0 0x7f942c0fd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.242+0000 7f9434200700 1 -- 172.21.15.27:0/1396663988 >> 172.21.15.27:0/1396663988 conn(0x7f942c0f8540 msgr2=0x7f942c0fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.242+0000 7f9434200700 1 -- 172.21.15.27:0/1396663988 shutdown_connections 2024-04-16T05:47:09.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.243+0000 7f9434200700 1 -- 172.21.15.27:0/1396663988 wait complete. 2024-04-16T05:47:09.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.243+0000 7f9434200700 1 Processor -- start 2024-04-16T05:47:09.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.244+0000 7f9434200700 1 -- start start 2024-04-16T05:47:09.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.244+0000 7f9434200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f942c0fcdc0 0x7f942c10fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:09.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.244+0000 7f9434200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f942c0fd710 0x7f942c1102a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:09.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.244+0000 7f9434200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942c1108a0 con 0x7f942c0fd710 2024-04-16T05:47:09.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.244+0000 7f9434200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942c110fe0 con 0x7f942c0fcdc0 2024-04-16T05:47:09.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.244+0000 7f9430e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f942c0fd710 0x7f942c1102a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:09.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.244+0000 7f9430e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f942c0fd710 0x7f942c1102a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41008/0 (socket says 172.21.15.27:41008) 2024-04-16T05:47:09.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.244+0000 7f9430e00700 1 -- 172.21.15.27:0/2447550208 learned_addr learned my addr 172.21.15.27:0/2447550208 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:47:09.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.245+0000 7f9431800700 1 --2- 172.21.15.27:0/2447550208 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f942c0fcdc0 0x7f942c10fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:09.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.245+0000 7f9430e00700 1 -- 172.21.15.27:0/2447550208 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f942c0fcdc0 msgr2=0x7f942c10fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:09.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.245+0000 7f9430e00700 1 --2- 172.21.15.27:0/2447550208 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f942c0fcdc0 0x7f942c10fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:09.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.245+0000 7f9430e00700 1 -- 172.21.15.27:0/2447550208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9418005040 con 0x7f942c0fd710 2024-04-16T05:47:09.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.245+0000 7f9430e00700 1 --2- 172.21.15.27:0/2447550208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f942c0fd710 0x7f942c1102a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f941800b970 tx=0x7f9418009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:09.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.246+0000 7f9429600700 1 -- 172.21.15.27:0/2447550208 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f941800e040 con 0x7f942c0fd710 2024-04-16T05:47:09.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.246+0000 7f9429600700 1 -- 172.21.15.27:0/2447550208 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9418004290 con 0x7f942c0fd710 2024-04-16T05:47:09.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.246+0000 7f9429600700 1 -- 172.21.15.27:0/2447550208 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f941801c630 con 0x7f942c0fd710 2024-04-16T05:47:09.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.246+0000 7f9434200700 1 -- 172.21.15.27:0/2447550208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f942c111260 con 0x7f942c0fd710 2024-04-16T05:47:09.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.246+0000 7f9434200700 1 -- 172.21.15.27:0/2447550208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f942c111820 con 0x7f942c0fd710 2024-04-16T05:47:09.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.247+0000 7f9434200700 1 -- 172.21.15.27:0/2447550208 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f942c101a60 con 0x7f942c0fd710 2024-04-16T05:47:09.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.250+0000 7f9429600700 1 -- 172.21.15.27:0/2447550208 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f941800bbf0 con 0x7f942c0fd710 2024-04-16T05:47:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.252+0000 7f9429600700 1 --2- 172.21.15.27:0/2447550208 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f941406ec10 0x7f94140710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.252+0000 7f9429600700 1 -- 172.21.15.27:0/2447550208 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f941808fba0 con 0x7f942c0fd710 2024-04-16T05:47:09.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.252+0000 7f9431800700 1 --2- 172.21.15.27:0/2447550208 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f941406ec10 0x7f94140710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:09.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.253+0000 7f9431800700 1 --2- 172.21.15.27:0/2447550208 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f941406ec10 0x7f94140710c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9420009fd0 tx=0x7f9420009f00 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:09.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.259+0000 7f9429600700 1 -- 172.21.15.27:0/2447550208 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f941805c500 con 0x7f942c0fd710 2024-04-16T05:47:09.407 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:09 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:09.407 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:09 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:09.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.524+0000 7f9434200700 1 -- 172.21.15.27:0/2447550208 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f942c111b50 con 0x7f941406ec10 2024-04-16T05:47:09.524 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:09 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:09.524 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:09 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:09.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.527+0000 7f9429600700 1 -- 172.21.15.27:0/2447550208 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+402 (secure 0 0 0) 0x7f942c111b50 con 0x7f941406ec10 2024-04-16T05:47:09.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.530+0000 7f9434200700 1 -- 172.21.15.27:0/2447550208 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f941406ec10 msgr2=0x7f94140710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:09.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.530+0000 7f9434200700 1 --2- 172.21.15.27:0/2447550208 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f941406ec10 0x7f94140710c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9420009fd0 tx=0x7f9420009f00 comp rx=0 tx=0).stop 2024-04-16T05:47:09.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.530+0000 7f9434200700 1 -- 172.21.15.27:0/2447550208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f942c0fd710 msgr2=0x7f942c1102a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:09.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.530+0000 7f9434200700 1 --2- 172.21.15.27:0/2447550208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f942c0fd710 0x7f942c1102a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f941800b970 tx=0x7f9418009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:09.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.530+0000 7f9434200700 1 -- 172.21.15.27:0/2447550208 shutdown_connections 2024-04-16T05:47:09.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.530+0000 7f9434200700 1 --2- 172.21.15.27:0/2447550208 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f941406ec10 0x7f94140710c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:09.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.530+0000 7f9434200700 1 --2- 172.21.15.27:0/2447550208 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f942c0fcdc0 0x7f942c10fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:09.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.530+0000 7f9434200700 1 --2- 172.21.15.27:0/2447550208 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f942c0fd710 0x7f942c1102a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:09.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.530+0000 7f9434200700 1 -- 172.21.15.27:0/2447550208 >> 172.21.15.27:0/2447550208 conn(0x7f942c0f8540 msgr2=0x7f942c0f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:09.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.530+0000 7f9434200700 1 -- 172.21.15.27:0/2447550208 shutdown_connections 2024-04-16T05:47:09.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.530+0000 7f9434200700 1 -- 172.21.15.27:0/2447550208 wait complete. 2024-04-16T05:47:09.546 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:47:09.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.708+0000 7fb86c000700 1 -- 172.21.15.27:0/121510287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb864070890 msgr2=0x7fb864070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:09.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.708+0000 7fb86c000700 1 --2- 172.21.15.27:0/121510287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb864070890 0x7fb864070d00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb850004940 tx=0x7fb850009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:09.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.709+0000 7fb86c000700 1 -- 172.21.15.27:0/121510287 shutdown_connections 2024-04-16T05:47:09.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.709+0000 7fb86c000700 1 --2- 172.21.15.27:0/121510287 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb864070890 0x7fb864070d00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:09.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.709+0000 7fb86c000700 1 --2- 172.21.15.27:0/121510287 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb864071e50 0x7fb864070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:09.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.709+0000 7fb86c000700 1 -- 172.21.15.27:0/121510287 >> 172.21.15.27:0/121510287 conn(0x7fb8640f8540 msgr2=0x7fb8640fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:09.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.709+0000 7fb86c000700 1 -- 172.21.15.27:0/121510287 shutdown_connections 2024-04-16T05:47:09.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.709+0000 7fb86c000700 1 -- 172.21.15.27:0/121510287 wait complete. 2024-04-16T05:47:09.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.710+0000 7fb86c000700 1 Processor -- start 2024-04-16T05:47:09.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.710+0000 7fb86c000700 1 -- start start 2024-04-16T05:47:09.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.711+0000 7fb86c000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb864070890 0x7fb86410fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:09.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.711+0000 7fb86c000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb864071e50 0x7fb864110190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:09.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.711+0000 7fb86c000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb864110790 con 0x7fb864070890 2024-04-16T05:47:09.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.711+0000 7fb86c000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb864110fe0 con 0x7fb864071e50 2024-04-16T05:47:09.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.711+0000 7fb869600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb864070890 0x7fb86410fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:09.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.711+0000 7fb869600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb864070890 0x7fb86410fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41038/0 (socket says 172.21.15.27:41038) 2024-04-16T05:47:09.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.711+0000 7fb869600700 1 -- 172.21.15.27:0/4286754759 learned_addr learned my addr 172.21.15.27:0/4286754759 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:47:09.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.711+0000 7fb868c00700 1 --2- 172.21.15.27:0/4286754759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb864071e50 0x7fb864110190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:09.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.711+0000 7fb869600700 1 -- 172.21.15.27:0/4286754759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb864071e50 msgr2=0x7fb864110190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:09.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.711+0000 7fb869600700 1 --2- 172.21.15.27:0/4286754759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb864071e50 0x7fb864110190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:09.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.712+0000 7fb869600700 1 -- 172.21.15.27:0/4286754759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb850005040 con 0x7fb864070890 2024-04-16T05:47:09.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.712+0000 7fb868c00700 1 --2- 172.21.15.27:0/4286754759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb864071e50 0x7fb864110190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T05:47:09.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.712+0000 7fb869600700 1 --2- 172.21.15.27:0/4286754759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb864070890 0x7fb86410fc50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb858002a80 tx=0x7fb858004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.712+0000 7fb861600700 1 -- 172.21.15.27:0/4286754759 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb858007cc0 con 0x7fb864070890 2024-04-16T05:47:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.712+0000 7fb861600700 1 -- 172.21.15.27:0/4286754759 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb85800f040 con 0x7fb864070890 2024-04-16T05:47:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.712+0000 7fb861600700 1 -- 172.21.15.27:0/4286754759 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb85800dad0 con 0x7fb864070890 2024-04-16T05:47:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.712+0000 7fb86c000700 1 -- 172.21.15.27:0/4286754759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8641112c0 con 0x7fb864070890 2024-04-16T05:47:09.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.713+0000 7fb86c000700 1 -- 172.21.15.27:0/4286754759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb864111800 con 0x7fb864070890 2024-04-16T05:47:09.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.714+0000 7fb86c000700 1 -- 172.21.15.27:0/4286754759 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb86405f5f0 con 0x7fb864070890 2024-04-16T05:47:09.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.715+0000 7fb861600700 1 -- 172.21.15.27:0/4286754759 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb85800dc30 con 0x7fb864070890 2024-04-16T05:47:09.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.716+0000 7fb861600700 1 --2- 172.21.15.27:0/4286754759 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb84c06ec10 0x7fb84c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:09.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.716+0000 7fb861600700 1 -- 172.21.15.27:0/4286754759 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fb85808e570 con 0x7fb864070890 2024-04-16T05:47:09.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.717+0000 7fb868c00700 1 --2- 172.21.15.27:0/4286754759 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb84c06ec10 0x7fb84c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:09.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.718+0000 7fb868c00700 1 --2- 172.21.15.27:0/4286754759 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb84c06ec10 0x7fb84c0710c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb85000b3b0 tx=0x7fb850009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:09.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.722+0000 7fb861600700 1 -- 172.21.15.27:0/4286754759 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb85805aed0 con 0x7fb864070890 2024-04-16T05:47:09.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.992+0000 7fb86c000700 1 -- 172.21.15.27:0/4286754759 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb864111c50 con 0x7fb84c06ec10 2024-04-16T05:47:09.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.995+0000 7fb861600700 1 -- 172.21.15.27:0/4286754759 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+402 (secure 0 0 0) 0x7fb864111c50 con 0x7fb84c06ec10 2024-04-16T05:47:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.998+0000 7fb86c000700 1 -- 172.21.15.27:0/4286754759 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb84c06ec10 msgr2=0x7fb84c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.998+0000 7fb86c000700 1 --2- 172.21.15.27:0/4286754759 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb84c06ec10 0x7fb84c0710c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb85000b3b0 tx=0x7fb850009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.998+0000 7fb86c000700 1 -- 172.21.15.27:0/4286754759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb864070890 msgr2=0x7fb86410fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.998+0000 7fb86c000700 1 --2- 172.21.15.27:0/4286754759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb864070890 0x7fb86410fc50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb858002a80 tx=0x7fb858004600 comp rx=0 tx=0).stop 2024-04-16T05:47:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.999+0000 7fb86c000700 1 -- 172.21.15.27:0/4286754759 shutdown_connections 2024-04-16T05:47:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.999+0000 7fb86c000700 1 --2- 172.21.15.27:0/4286754759 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb84c06ec10 0x7fb84c0710c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.999+0000 7fb86c000700 1 --2- 172.21.15.27:0/4286754759 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb864070890 0x7fb86410fc50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.999+0000 7fb86c000700 1 --2- 172.21.15.27:0/4286754759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb864071e50 0x7fb864110190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.999+0000 7fb86c000700 1 -- 172.21.15.27:0/4286754759 >> 172.21.15.27:0/4286754759 conn(0x7fb8640f8540 msgr2=0x7fb8640f9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.999+0000 7fb86c000700 1 -- 172.21.15.27:0/4286754759 shutdown_connections 2024-04-16T05:47:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:09.999+0000 7fb86c000700 1 -- 172.21.15.27:0/4286754759 wait complete. 2024-04-16T05:47:10.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.159+0000 7ff06f200700 1 -- 172.21.15.27:0/3085717621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0680fff70 msgr2=0x7ff068102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:10.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.159+0000 7ff06f200700 1 --2- 172.21.15.27:0/3085717621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0680fff70 0x7ff068102390 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff050004940 tx=0x7ff050009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:10.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.159+0000 7ff06f200700 1 -- 172.21.15.27:0/3085717621 shutdown_connections 2024-04-16T05:47:10.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.159+0000 7ff06f200700 1 --2- 172.21.15.27:0/3085717621 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0680fff70 0x7ff068102390 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:10.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.159+0000 7ff06f200700 1 --2- 172.21.15.27:0/3085717621 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0680662f0 0x7ff0680ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.159+0000 7ff06f200700 1 -- 172.21.15.27:0/3085717621 >> 172.21.15.27:0/3085717621 conn(0x7ff0680f74c0 msgr2=0x7ff0680f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.159+0000 7ff06f200700 1 -- 172.21.15.27:0/3085717621 shutdown_connections 2024-04-16T05:47:10.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.159+0000 7ff06f200700 1 -- 172.21.15.27:0/3085717621 wait complete. 2024-04-16T05:47:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.160+0000 7ff06f200700 1 Processor -- start 2024-04-16T05:47:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.160+0000 7ff06f200700 1 -- start start 2024-04-16T05:47:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.161+0000 7ff06f200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0680662f0 0x7ff06810fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.161+0000 7ff06f200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0680fff70 0x7ff0681101e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.161+0000 7ff06f200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff068110870 con 0x7ff0680662f0 2024-04-16T05:47:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.161+0000 7ff06f200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff068111010 con 0x7ff0680fff70 2024-04-16T05:47:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.161+0000 7ff067e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0680662f0 0x7ff06810fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.161+0000 7ff067e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0680662f0 0x7ff06810fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41050/0 (socket says 172.21.15.27:41050) 2024-04-16T05:47:10.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.161+0000 7ff067e00700 1 -- 172.21.15.27:0/99962422 learned_addr learned my addr 172.21.15.27:0/99962422 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:47:10.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.161+0000 7ff05fe00700 1 --2- 172.21.15.27:0/99962422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0680fff70 0x7ff0681101e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:10.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.162+0000 7ff067e00700 1 -- 172.21.15.27:0/99962422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0680fff70 msgr2=0x7ff0681101e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:10.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.162+0000 7ff067e00700 1 --2- 172.21.15.27:0/99962422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0680fff70 0x7ff0681101e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:10.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.162+0000 7ff067e00700 1 -- 172.21.15.27:0/99962422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff050005040 con 0x7ff0680662f0 2024-04-16T05:47:10.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.162+0000 7ff067e00700 1 --2- 172.21.15.27:0/99962422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0680662f0 0x7ff06810fca0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff05800b790 tx=0x7ff058009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:10.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.162+0000 7ff064c00700 1 -- 172.21.15.27:0/99962422 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff05800f6f0 con 0x7ff0680662f0 2024-04-16T05:47:10.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.162+0000 7ff064c00700 1 -- 172.21.15.27:0/99962422 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff05800fd20 con 0x7ff0680662f0 2024-04-16T05:47:10.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.162+0000 7ff06f200700 1 -- 172.21.15.27:0/99962422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0681112f0 con 0x7ff0680662f0 2024-04-16T05:47:10.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.162+0000 7ff064c00700 1 -- 172.21.15.27:0/99962422 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0580103a0 con 0x7ff0680662f0 2024-04-16T05:47:10.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.163+0000 7ff06f200700 1 -- 172.21.15.27:0/99962422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0681118b0 con 0x7ff0680662f0 2024-04-16T05:47:10.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.164+0000 7ff06f200700 1 -- 172.21.15.27:0/99962422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff06805f5f0 con 0x7ff0680662f0 2024-04-16T05:47:10.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.165+0000 7ff064c00700 1 -- 172.21.15.27:0/99962422 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff058011400 con 0x7ff0680662f0 2024-04-16T05:47:10.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.166+0000 7ff064c00700 1 --2- 172.21.15.27:0/99962422 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff04c06ec10 0x7ff04c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:10.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.166+0000 7ff064c00700 1 -- 172.21.15.27:0/99962422 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7ff05808e7c0 con 0x7ff0680662f0 2024-04-16T05:47:10.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.167+0000 7ff05fe00700 1 --2- 172.21.15.27:0/99962422 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff04c06ec10 0x7ff04c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:10.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.167+0000 7ff05fe00700 1 --2- 172.21.15.27:0/99962422 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff04c06ec10 0x7ff04c0710c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff050004f90 tx=0x7ff050009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:10.170 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.171+0000 7ff064c00700 1 -- 172.21.15.27:0/99962422 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff05805b020 con 0x7ff0680662f0 2024-04-16T05:47:10.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.408+0000 7ff06f200700 1 -- 172.21.15.27:0/99962422 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff068111c40 con 0x7ff04c06ec10 2024-04-16T05:47:10.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.420+0000 7ff064c00700 1 -- 172.21.15.27:0/99962422 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7ff068111c40 con 0x7ff04c06ec10 2024-04-16T05:47:10.420 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:47:10.420 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (3m) 2s ago 14m 13.6M - 0.25.0 c8568f914cd2 4ed63a8a6b33 2024-04-16T05:47:10.420 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (17s) 2s ago 14m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 4aa1f0006685 2024-04-16T05:47:10.420 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (10s) 5s ago 13m 6656k - 19.0.0-2884-g218a6914 a4e4811815e4 1c1912507d56 2024-04-16T05:47:10.420 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (3m) 2s ago 13m 75.4M - 9.4.7 954c08fa6188 f7ee2a1afd12 2024-04-16T05:47:10.420 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (6m) 2s ago 6m 23.7M - 17.2.7-847-g45317f95 b333581ccdf5 0824e2621be5 2024-04-16T05:47:10.420 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (6m) 2s ago 6m 13.6M - 17.2.7-847-g45317f95 b333581ccdf5 b332a72cc594 2024-04-16T05:47:10.420 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (6m) 5s ago 6m 13.6M - 17.2.7-847-g45317f95 b333581ccdf5 f5e36815ddfa 2024-04-16T05:47:10.420 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (6m) 5s ago 6m 13.5M - 17.2.7-847-g45317f95 b333581ccdf5 755a73c65a7f 2024-04-16T05:47:10.420 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:8443,9283,8765 running (3m) 2s ago 15m 496M - 19.0.0-2884-g218a6914 a4e4811815e4 b69a0b08e12b 2024-04-16T05:47:10.420 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283,8765 running (2m) 5s ago 12m 451M - 19.0.0-2884-g218a6914 a4e4811815e4 32e27bcd7e42 2024-04-16T05:47:10.420 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (111s) 2s ago 16m 40.5M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 299ca337e4ca 2024-04-16T05:47:10.421 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (78s) 5s ago 12m 31.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5db5f3037ed0 2024-04-16T05:47:10.421 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (3m) 2s ago 14m 7964k - 1.5.0 0da6a335fe13 5e29e29868b0 2024-04-16T05:47:10.421 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (3m) 5s ago 12m 8032k - 1.5.0 0da6a335fe13 a360922bc621 2024-04-16T05:47:10.421 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (11m) 2s ago 11m 47.1M 1557M 17.2.7-847-g45317f95 b333581ccdf5 15e049573bca 2024-04-16T05:47:10.421 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (11m) 2s ago 11m 48.0M 1557M 17.2.7-847-g45317f95 b333581ccdf5 e146295f6500 2024-04-16T05:47:10.421 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (10m) 2s ago 10m 49.3M 1557M 17.2.7-847-g45317f95 b333581ccdf5 6212f6a18fa4 2024-04-16T05:47:10.421 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (9m) 5s ago 9m 46.3M 2581M 17.2.7-847-g45317f95 b333581ccdf5 d6724ad05f9f 2024-04-16T05:47:10.421 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (9m) 5s ago 9m 45.0M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7a59908a3e10 2024-04-16T05:47:10.421 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (8m) 5s ago 8m 44.2M 2581M 17.2.7-847-g45317f95 b333581ccdf5 8e4b60163907 2024-04-16T05:47:10.421 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (2m) 2s ago 13m 54.4M - 2.43.0 a07b618ecd1d e70f55ba5ec0 2024-04-16T05:47:10.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.423+0000 7ff06f200700 1 -- 172.21.15.27:0/99962422 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff04c06ec10 msgr2=0x7ff04c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:10.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.423+0000 7ff06f200700 1 --2- 172.21.15.27:0/99962422 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff04c06ec10 0x7ff04c0710c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff050004f90 tx=0x7ff050009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.423+0000 7ff06f200700 1 -- 172.21.15.27:0/99962422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0680662f0 msgr2=0x7ff06810fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.423+0000 7ff06f200700 1 --2- 172.21.15.27:0/99962422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0680662f0 0x7ff06810fca0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff05800b790 tx=0x7ff058009220 comp rx=0 tx=0).stop 2024-04-16T05:47:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.423+0000 7ff06f200700 1 -- 172.21.15.27:0/99962422 shutdown_connections 2024-04-16T05:47:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.423+0000 7ff06f200700 1 --2- 172.21.15.27:0/99962422 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff04c06ec10 0x7ff04c0710c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.423+0000 7ff06f200700 1 --2- 172.21.15.27:0/99962422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff0680662f0 0x7ff06810fca0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.423+0000 7ff06f200700 1 --2- 172.21.15.27:0/99962422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff0680fff70 0x7ff0681101e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.423+0000 7ff06f200700 1 -- 172.21.15.27:0/99962422 >> 172.21.15.27:0/99962422 conn(0x7ff0680f74c0 msgr2=0x7ff0680fbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.424+0000 7ff06f200700 1 -- 172.21.15.27:0/99962422 shutdown_connections 2024-04-16T05:47:10.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.424+0000 7ff06f200700 1 -- 172.21.15.27:0/99962422 wait complete. 2024-04-16T05:47:10.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.582+0000 7fa5a1c00700 1 -- 172.21.15.27:0/2554233217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa59c0feff0 msgr2=0x7fa59c0ff400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:10.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.582+0000 7fa5a1c00700 1 --2- 172.21.15.27:0/2554233217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa59c0feff0 0x7fa59c0ff400 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa58c004940 tx=0x7fa58c009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:10.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.583+0000 7fa5a1c00700 1 -- 172.21.15.27:0/2554233217 shutdown_connections 2024-04-16T05:47:10.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.583+0000 7fa5a1c00700 1 --2- 172.21.15.27:0/2554233217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa59c1002a0 0x7fa59c100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:10.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.583+0000 7fa5a1c00700 1 --2- 172.21.15.27:0/2554233217 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa59c0feff0 0x7fa59c0ff400 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:10.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.583+0000 7fa5a1c00700 1 -- 172.21.15.27:0/2554233217 >> 172.21.15.27:0/2554233217 conn(0x7fa59c0fa790 msgr2=0x7fa59c0fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:10.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.583+0000 7fa5a1c00700 1 -- 172.21.15.27:0/2554233217 shutdown_connections 2024-04-16T05:47:10.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.583+0000 7fa5a1c00700 1 -- 172.21.15.27:0/2554233217 wait complete. 2024-04-16T05:47:10.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.584+0000 7fa5a1c00700 1 Processor -- start 2024-04-16T05:47:10.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.584+0000 7fa5a1c00700 1 -- start start 2024-04-16T05:47:10.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.584+0000 7fa5a1c00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa59c0feff0 0x7fa59c10fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:10.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.584+0000 7fa5a1c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa59c1002a0 0x7fa59c110050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:10.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.584+0000 7fa5a1c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c110650 con 0x7fa59c1002a0 2024-04-16T05:47:10.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.584+0000 7fa5a1c00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59c1107c0 con 0x7fa59c0feff0 2024-04-16T05:47:10.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.584+0000 7fa59a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa59c1002a0 0x7fa59c110050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:10.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.584+0000 7fa59a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa59c1002a0 0x7fa59c110050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41062/0 (socket says 172.21.15.27:41062) 2024-04-16T05:47:10.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.584+0000 7fa59a000700 1 -- 172.21.15.27:0/2966384836 learned_addr learned my addr 172.21.15.27:0/2966384836 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:47:10.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.585+0000 7fa59aa00700 1 --2- 172.21.15.27:0/2966384836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa59c0feff0 0x7fa59c10fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:10.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.585+0000 7fa59a000700 1 -- 172.21.15.27:0/2966384836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa59c0feff0 msgr2=0x7fa59c10fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:10.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.585+0000 7fa59a000700 1 --2- 172.21.15.27:0/2966384836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa59c0feff0 0x7fa59c10fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:10.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.585+0000 7fa59a000700 1 -- 172.21.15.27:0/2966384836 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58c005040 con 0x7fa59c1002a0 2024-04-16T05:47:10.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.585+0000 7fa59a000700 1 --2- 172.21.15.27:0/2966384836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa59c1002a0 0x7fa59c110050 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa588002a50 tx=0x7fa5880046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:10.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.586+0000 7fa592a00700 1 -- 172.21.15.27:0/2966384836 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa588007cc0 con 0x7fa59c1002a0 2024-04-16T05:47:10.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.586+0000 7fa592a00700 1 -- 172.21.15.27:0/2966384836 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa58800f040 con 0x7fa59c1002a0 2024-04-16T05:47:10.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.586+0000 7fa5a1c00700 1 -- 172.21.15.27:0/2966384836 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa59c115220 con 0x7fa59c1002a0 2024-04-16T05:47:10.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.586+0000 7fa592a00700 1 -- 172.21.15.27:0/2966384836 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa58800da30 con 0x7fa59c1002a0 2024-04-16T05:47:10.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.586+0000 7fa5a1c00700 1 -- 172.21.15.27:0/2966384836 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa59c1157e0 con 0x7fa59c1002a0 2024-04-16T05:47:10.586 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.587+0000 7fa5a1c00700 1 -- 172.21.15.27:0/2966384836 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa59c04c710 con 0x7fa59c1002a0 2024-04-16T05:47:10.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.589+0000 7fa592a00700 1 -- 172.21.15.27:0/2966384836 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa58800db90 con 0x7fa59c1002a0 2024-04-16T05:47:10.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.590+0000 7fa592a00700 1 --2- 172.21.15.27:0/2966384836 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa58006ec00 0x7fa5800710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:10.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.590+0000 7fa592a00700 1 -- 172.21.15.27:0/2966384836 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fa58808e6a0 con 0x7fa59c1002a0 2024-04-16T05:47:10.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.590+0000 7fa59aa00700 1 --2- 172.21.15.27:0/2966384836 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa58006ec00 0x7fa5800710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:10.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.591+0000 7fa59aa00700 1 --2- 172.21.15.27:0/2966384836 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa58006ec00 0x7fa5800710b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa58c00b970 tx=0x7fa58c009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:10.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.596+0000 7fa592a00700 1 -- 172.21.15.27:0/2966384836 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa58805af00 con 0x7fa59c1002a0 2024-04-16T05:47:10.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:10 smithi122 ceph-mon[80930]: pgmap v53: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T05:47:10.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:10 smithi122 ceph-mon[80930]: from='client.34212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:47:10.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:10 smithi027 ceph-mon[93635]: pgmap v53: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T05:47:10.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:10 smithi027 ceph-mon[93635]: from='client.34212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:47:10.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.910+0000 7fa5a1c00700 1 -- 172.21.15.27:0/2966384836 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fa59c115c10 con 0x7fa59c1002a0 2024-04-16T05:47:10.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.910+0000 7fa592a00700 1 -- 172.21.15.27:0/2966384836 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+736 (secure 0 0 0) 0x7fa58805aa90 con 0x7fa59c1002a0 2024-04-16T05:47:10.910 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:47:10.910 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:47:10.910 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:47:10.910 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:47:10.910 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:47:10.910 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:47:10.910 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:47:10.910 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:47:10.910 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6 2024-04-16T05:47:10.910 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:47:10.910 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:47:10.910 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T05:47:10.910 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:47:10.911 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:47:10.911 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 10, 2024-04-16T05:47:10.911 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T05:47:10.911 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:47:10.911 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:47:10.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.913+0000 7fa5a1c00700 1 -- 172.21.15.27:0/2966384836 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa58006ec00 msgr2=0x7fa5800710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:10.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.913+0000 7fa5a1c00700 1 --2- 172.21.15.27:0/2966384836 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa58006ec00 0x7fa5800710b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa58c00b970 tx=0x7fa58c009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:10.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.913+0000 7fa5a1c00700 1 -- 172.21.15.27:0/2966384836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa59c1002a0 msgr2=0x7fa59c110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:10.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.913+0000 7fa5a1c00700 1 --2- 172.21.15.27:0/2966384836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa59c1002a0 0x7fa59c110050 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa588002a50 tx=0x7fa5880046b0 comp rx=0 tx=0).stop 2024-04-16T05:47:10.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.913+0000 7fa5a1c00700 1 -- 172.21.15.27:0/2966384836 shutdown_connections 2024-04-16T05:47:10.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.913+0000 7fa5a1c00700 1 --2- 172.21.15.27:0/2966384836 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa58006ec00 0x7fa5800710b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:10.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.913+0000 7fa5a1c00700 1 --2- 172.21.15.27:0/2966384836 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa59c0feff0 0x7fa59c10fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:10.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.913+0000 7fa5a1c00700 1 --2- 172.21.15.27:0/2966384836 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa59c1002a0 0x7fa59c110050 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:10.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.913+0000 7fa5a1c00700 1 -- 172.21.15.27:0/2966384836 >> 172.21.15.27:0/2966384836 conn(0x7fa59c0fa790 msgr2=0x7fa59c0fbe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:10.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.913+0000 7fa5a1c00700 1 -- 172.21.15.27:0/2966384836 shutdown_connections 2024-04-16T05:47:10.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:10.914+0000 7fa5a1c00700 1 -- 172.21.15.27:0/2966384836 wait complete. 2024-04-16T05:47:11.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.062+0000 7f20e4a00700 1 -- 172.21.15.27:0/1519621483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20e0066270 msgr2=0x7f20e00ffab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:11.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.062+0000 7f20e4a00700 1 --2- 172.21.15.27:0/1519621483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20e0066270 0x7f20e00ffab0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f20d0004940 tx=0x7f20d0009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:11.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.062+0000 7f20e4a00700 1 -- 172.21.15.27:0/1519621483 shutdown_connections 2024-04-16T05:47:11.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.062+0000 7f20e4a00700 1 --2- 172.21.15.27:0/1519621483 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20e00ffff0 0x7f20e01023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:11.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.062+0000 7f20e4a00700 1 --2- 172.21.15.27:0/1519621483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20e0066270 0x7f20e00ffab0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:11.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.062+0000 7f20e4a00700 1 -- 172.21.15.27:0/1519621483 >> 172.21.15.27:0/1519621483 conn(0x7f20e00f74c0 msgr2=0x7f20e00f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:11.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.062+0000 7f20e4a00700 1 -- 172.21.15.27:0/1519621483 shutdown_connections 2024-04-16T05:47:11.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.062+0000 7f20e4a00700 1 -- 172.21.15.27:0/1519621483 wait complete. 2024-04-16T05:47:11.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.063+0000 7f20e4a00700 1 Processor -- start 2024-04-16T05:47:11.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.064+0000 7f20e4a00700 1 -- start start 2024-04-16T05:47:11.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.064+0000 7f20e4a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20e0066270 0x7f20e00fd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:11.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.064+0000 7f20e4a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20e00ffff0 0x7f20e00fb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:11.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.064+0000 7f20e4a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e00fbfd0 con 0x7f20e00ffff0 2024-04-16T05:47:11.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.064+0000 7f20e4a00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e00fc110 con 0x7f20e0066270 2024-04-16T05:47:11.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.064+0000 7f20dd400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20e00ffff0 0x7f20e00fb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:11.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.065+0000 7f20dd400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20e00ffff0 0x7f20e00fb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41092/0 (socket says 172.21.15.27:41092) 2024-04-16T05:47:11.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.065+0000 7f20dd400700 1 -- 172.21.15.27:0/3695080600 learned_addr learned my addr 172.21.15.27:0/3695080600 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:47:11.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.065+0000 7f20dde00700 1 --2- 172.21.15.27:0/3695080600 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20e0066270 0x7f20e00fd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:11.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.065+0000 7f20dd400700 1 -- 172.21.15.27:0/3695080600 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20e0066270 msgr2=0x7f20e00fd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:11.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.065+0000 7f20dd400700 1 --2- 172.21.15.27:0/3695080600 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20e0066270 0x7f20e00fd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:11.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.065+0000 7f20dd400700 1 -- 172.21.15.27:0/3695080600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d0005040 con 0x7f20e00ffff0 2024-04-16T05:47:11.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.065+0000 7f20dd400700 1 --2- 172.21.15.27:0/3695080600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20e00ffff0 0x7f20e00fb9b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f20cc00b790 tx=0x7f20cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:11.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.066+0000 7f20d6000700 1 -- 172.21.15.27:0/3695080600 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20cc00f6f0 con 0x7f20e00ffff0 2024-04-16T05:47:11.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.066+0000 7f20d6000700 1 -- 172.21.15.27:0/3695080600 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f20cc00fd20 con 0x7f20e00ffff0 2024-04-16T05:47:11.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.066+0000 7f20e4a00700 1 -- 172.21.15.27:0/3695080600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e00fc3f0 con 0x7f20e00ffff0 2024-04-16T05:47:11.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.066+0000 7f20d6000700 1 -- 172.21.15.27:0/3695080600 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20cc0103a0 con 0x7f20e00ffff0 2024-04-16T05:47:11.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.066+0000 7f20e4a00700 1 -- 172.21.15.27:0/3695080600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e0115af0 con 0x7f20e00ffff0 2024-04-16T05:47:11.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.067+0000 7f20e4a00700 1 -- 172.21.15.27:0/3695080600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e005f5f0 con 0x7f20e00ffff0 2024-04-16T05:47:11.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.068+0000 7f20d6000700 1 -- 172.21.15.27:0/3695080600 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f20cc011400 con 0x7f20e00ffff0 2024-04-16T05:47:11.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.069+0000 7f20d6000700 1 --2- 172.21.15.27:0/3695080600 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f20c406ec10 0x7f20c40710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:11.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.069+0000 7f20d6000700 1 -- 172.21.15.27:0/3695080600 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f20cc08e6b0 con 0x7f20e00ffff0 2024-04-16T05:47:11.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.069+0000 7f20dde00700 1 --2- 172.21.15.27:0/3695080600 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f20c406ec10 0x7f20c40710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:11.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.070+0000 7f20dde00700 1 --2- 172.21.15.27:0/3695080600 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f20c406ec10 0x7f20c40710c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f20d0000c00 tx=0x7f20d0009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:11.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.074+0000 7f20d6000700 1 -- 172.21.15.27:0/3695080600 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f20cc0574e0 con 0x7f20e00ffff0 2024-04-16T05:47:11.254 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 292, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86663 2024-04-16T05:47:11.255 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 292, success, outsize: 120 2024-04-16T05:47:11.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 294, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86747 2024-04-16T05:47:11.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 294, success, outsize: 120 2024-04-16T05:47:11.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 296, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86834 2024-04-16T05:47:11.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 296, success, outsize: 120 2024-04-16T05:47:11.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 298, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 86927 2024-04-16T05:47:11.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 298, success, outsize: 120 2024-04-16T05:47:11.364 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 300, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87017 2024-04-16T05:47:11.365 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 300, success, outsize: 120 2024-04-16T05:47:11.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 302, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87101 2024-04-16T05:47:11.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 302, success, outsize: 120 2024-04-16T05:47:11.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.396+0000 7f20e4a00700 1 -- 172.21.15.27:0/3695080600 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f20e004bed0 con 0x7f20e00ffff0 2024-04-16T05:47:11.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.397+0000 7f20d6000700 1 -- 172.21.15.27:0/3695080600 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1880 (secure 0 0 0) 0x7f20cc05aba0 con 0x7f20e00ffff0 2024-04-16T05:47:11.397 INFO:teuthology.orchestra.run.smithi027.stdout:e16 2024-04-16T05:47:11.397 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:47:11.397 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:47:11.397 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:47:11.397 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:47:11.397 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:47:11.398 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:47:11.398 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 15 2024-04-16T05:47:11.398 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:47:11.398 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:47:11.398 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:40:32.701649+0000 2024-04-16T05:47:11.398 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:47:11.398 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:47:11.398 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:47:11.398 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:47:11.398 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:47:11.398 INFO:teuthology.orchestra.run.smithi027.stdout:max_xattr_size 65536 2024-04-16T05:47:11.398 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:47:11.398 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:47:11.398 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 46 2024-04-16T05:47:11.398 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:47:11.399 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:47:11.399 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:47:11.399 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=14462} 2024-04-16T05:47:11.399 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:47:11.399 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:47:11.399 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:47:11.399 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:47:11.399 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:47:11.399 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:47:11.399 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:47:11.399 INFO:teuthology.orchestra.run.smithi027.stdout:bal_rank_mask -1 2024-04-16T05:47:11.399 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:47:11.399 INFO:teuthology.orchestra.run.smithi027.stdout:qdb_cluster leader: 0 members: 2024-04-16T05:47:11.399 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:47:11.399 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:47:11.400 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:47:11.400 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:47:11.400 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:47:11.400 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{-1:14474} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2065690722,v1:172.21.15.27:6827/2065690722] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:47:11.400 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:24263} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:47:11.400 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:24273} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:47:11.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.401+0000 7f20e4a00700 1 -- 172.21.15.27:0/3695080600 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f20c406ec10 msgr2=0x7f20c40710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:11.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.401+0000 7f20e4a00700 1 --2- 172.21.15.27:0/3695080600 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f20c406ec10 0x7f20c40710c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f20d0000c00 tx=0x7f20d0009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:11.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.401+0000 7f20e4a00700 1 -- 172.21.15.27:0/3695080600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20e00ffff0 msgr2=0x7f20e00fb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:11.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.401+0000 7f20e4a00700 1 --2- 172.21.15.27:0/3695080600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20e00ffff0 0x7f20e00fb9b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f20cc00b790 tx=0x7f20cc009250 comp rx=0 tx=0).stop 2024-04-16T05:47:11.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.402+0000 7f20e4a00700 1 -- 172.21.15.27:0/3695080600 shutdown_connections 2024-04-16T05:47:11.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.402+0000 7f20e4a00700 1 --2- 172.21.15.27:0/3695080600 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f20c406ec10 0x7f20c40710c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:11.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.402+0000 7f20e4a00700 1 --2- 172.21.15.27:0/3695080600 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f20e0066270 0x7f20e00fd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:11.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.402+0000 7f20e4a00700 1 --2- 172.21.15.27:0/3695080600 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f20e00ffff0 0x7f20e00fb9b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:11.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.402+0000 7f20e4a00700 1 -- 172.21.15.27:0/3695080600 >> 172.21.15.27:0/3695080600 conn(0x7f20e00f74c0 msgr2=0x7f20e00f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:11.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.402+0000 7f20e4a00700 1 -- 172.21.15.27:0/3695080600 shutdown_connections 2024-04-16T05:47:11.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.402+0000 7f20e4a00700 1 -- 172.21.15.27:0/3695080600 wait complete. 2024-04-16T05:47:11.402 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2024-04-16T05:47:11.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 304, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87188 2024-04-16T05:47:11.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 304, success, outsize: 120 2024-04-16T05:47:11.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 306, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87281 2024-04-16T05:47:11.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 306, success, outsize: 120 2024-04-16T05:47:11.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 308, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87371 2024-04-16T05:47:11.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 308, success, outsize: 120 2024-04-16T05:47:11.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 310, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87455 2024-04-16T05:47:11.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 310, success, outsize: 120 2024-04-16T05:47:11.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 312, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87542 2024-04-16T05:47:11.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 312, success, outsize: 120 2024-04-16T05:47:11.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 314, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87635 2024-04-16T05:47:11.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 314, success, outsize: 120 2024-04-16T05:47:11.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.552+0000 7f9752e00700 1 -- 172.21.15.27:0/2036401266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f974c0fedc0 msgr2=0x7f974c0faf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:11.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.552+0000 7f9752e00700 1 --2- 172.21.15.27:0/2036401266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f974c0fedc0 0x7f974c0faf00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9734004940 tx=0x7f9734009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:11.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.552+0000 7f9752e00700 1 -- 172.21.15.27:0/2036401266 shutdown_connections 2024-04-16T05:47:11.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.552+0000 7f9752e00700 1 --2- 172.21.15.27:0/2036401266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f974c0fedc0 0x7f974c0faf00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:11.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.552+0000 7f9752e00700 1 --2- 172.21.15.27:0/2036401266 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f974c0fe470 0x7f974c0fe880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:11.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.552+0000 7f9752e00700 1 -- 172.21.15.27:0/2036401266 >> 172.21.15.27:0/2036401266 conn(0x7f974c0f6c10 msgr2=0x7f974c0f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:11.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.552+0000 7f9752e00700 1 -- 172.21.15.27:0/2036401266 shutdown_connections 2024-04-16T05:47:11.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.552+0000 7f9752e00700 1 -- 172.21.15.27:0/2036401266 wait complete. 2024-04-16T05:47:11.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.553+0000 7f9752e00700 1 Processor -- start 2024-04-16T05:47:11.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.553+0000 7f9752e00700 1 -- start start 2024-04-16T05:47:11.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.554+0000 7f9752e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f974c0fe470 0x7f974c0fda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:11.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.554+0000 7f9752e00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f974c0fedc0 0x7f974c0fc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:11.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.554+0000 7f9752e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c0fe080 con 0x7f974c0fe470 2024-04-16T05:47:11.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.554+0000 7f9752e00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c0fc670 con 0x7f974c0fedc0 2024-04-16T05:47:11.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.554+0000 7f974be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f974c0fe470 0x7f974c0fda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:11.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.554+0000 7f974be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f974c0fe470 0x7f974c0fda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41112/0 (socket says 172.21.15.27:41112) 2024-04-16T05:47:11.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.554+0000 7f974be00700 1 -- 172.21.15.27:0/3108455949 learned_addr learned my addr 172.21.15.27:0/3108455949 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:47:11.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.554+0000 7f974b400700 1 --2- 172.21.15.27:0/3108455949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f974c0fedc0 0x7f974c0fc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:11.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.555+0000 7f974be00700 1 -- 172.21.15.27:0/3108455949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f974c0fedc0 msgr2=0x7f974c0fc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:11.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.555+0000 7f974be00700 1 --2- 172.21.15.27:0/3108455949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f974c0fedc0 0x7f974c0fc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:11.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.555+0000 7f974be00700 1 -- 172.21.15.27:0/3108455949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9734005040 con 0x7f974c0fe470 2024-04-16T05:47:11.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 316, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87725 2024-04-16T05:47:11.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 316, success, outsize: 120 2024-04-16T05:47:11.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.555+0000 7f974be00700 1 --2- 172.21.15.27:0/3108455949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f974c0fe470 0x7f974c0fda80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f973c002a50 tx=0x7f973c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:11.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.555+0000 7f9743e00700 1 -- 172.21.15.27:0/3108455949 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f973c007cc0 con 0x7f974c0fe470 2024-04-16T05:47:11.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.555+0000 7f9743e00700 1 -- 172.21.15.27:0/3108455949 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f973c00f040 con 0x7f974c0fe470 2024-04-16T05:47:11.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.555+0000 7f9752e00700 1 -- 172.21.15.27:0/3108455949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f974c0fc920 con 0x7f974c0fe470 2024-04-16T05:47:11.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.555+0000 7f9743e00700 1 -- 172.21.15.27:0/3108455949 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f973c00da30 con 0x7f974c0fe470 2024-04-16T05:47:11.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.555+0000 7f9752e00700 1 -- 172.21.15.27:0/3108455949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974c0fcee0 con 0x7f974c0fe470 2024-04-16T05:47:11.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.557+0000 7f9752e00700 1 -- 172.21.15.27:0/3108455949 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974c04c710 con 0x7f974c0fe470 2024-04-16T05:47:11.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.558+0000 7f9743e00700 1 -- 172.21.15.27:0/3108455949 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f973c00db90 con 0x7f974c0fe470 2024-04-16T05:47:11.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.558+0000 7f9743e00700 1 --2- 172.21.15.27:0/3108455949 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f973006ec00 0x7f97300710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:11.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.559+0000 7f9743e00700 1 -- 172.21.15.27:0/3108455949 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7f973c08e550 con 0x7f974c0fe470 2024-04-16T05:47:11.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.559+0000 7f974b400700 1 --2- 172.21.15.27:0/3108455949 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f973006ec00 0x7f97300710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:11.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.559+0000 7f974b400700 1 --2- 172.21.15.27:0/3108455949 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f973006ec00 0x7f97300710b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f973400b020 tx=0x7f9734009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:11.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.563+0000 7f9743e00700 1 -- 172.21.15.27:0/3108455949 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f973c05aeb0 con 0x7f974c0fe470 2024-04-16T05:47:11.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 318, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87809 2024-04-16T05:47:11.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 318, success, outsize: 120 2024-04-16T05:47:11.595 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 320, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87896 2024-04-16T05:47:11.595 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 320, success, outsize: 120 2024-04-16T05:47:11.617 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 322, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 87989 2024-04-16T05:47:11.617 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 322, success, outsize: 120 2024-04-16T05:47:11.638 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 324, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88079 2024-04-16T05:47:11.638 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 324, success, outsize: 120 2024-04-16T05:47:11.658 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 326, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88163 2024-04-16T05:47:11.658 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 326, success, outsize: 120 2024-04-16T05:47:11.668 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:11 smithi027 ceph-mon[93635]: from='client.34216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:47:11.668 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:11 smithi027 ceph-mon[93635]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:47:11.668 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:11 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:11.668 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:11 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:11.668 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:11 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/2966384836' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:47:11.668 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:11 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/3695080600' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:47:11.679 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 328, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88250 2024-04-16T05:47:11.680 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 328, success, outsize: 120 2024-04-16T05:47:11.701 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 330, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88343 2024-04-16T05:47:11.701 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 330, success, outsize: 120 2024-04-16T05:47:11.722 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 332, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88433 2024-04-16T05:47:11.722 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 332, success, outsize: 120 2024-04-16T05:47:11.742 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 334, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88517 2024-04-16T05:47:11.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 334, success, outsize: 120 2024-04-16T05:47:11.763 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 336, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88604 2024-04-16T05:47:11.764 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 336, success, outsize: 120 2024-04-16T05:47:11.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:11 smithi122 ceph-mon[80930]: from='client.34216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:47:11.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:11 smithi122 ceph-mon[80930]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:47:11.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:11 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:11.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:11 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:11.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:11 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/2966384836' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:47:11.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:11 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/3695080600' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:47:11.787 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 338, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88697 2024-04-16T05:47:11.787 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 338, success, outsize: 120 2024-04-16T05:47:11.809 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 340, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88787 2024-04-16T05:47:11.810 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 340, success, outsize: 120 2024-04-16T05:47:11.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.813+0000 7f9752e00700 1 -- 172.21.15.27:0/3108455949 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f974c102580 con 0x7f973006ec00 2024-04-16T05:47:11.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.815+0000 7f9743e00700 1 -- 172.21.15.27:0/3108455949 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+402 (secure 0 0 0) 0x7f974c102580 con 0x7f973006ec00 2024-04-16T05:47:11.813 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:47:11.814 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T05:47:11.814 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:47:11.814 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:47:11.814 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [ 2024-04-16T05:47:11.814 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr", 2024-04-16T05:47:11.814 INFO:teuthology.orchestra.run.smithi027.stdout: "mon", 2024-04-16T05:47:11.814 INFO:teuthology.orchestra.run.smithi027.stdout: "crash" 2024-04-16T05:47:11.814 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-16T05:47:11.814 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "6/21 daemons upgraded", 2024-04-16T05:47:11.814 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Currently upgrading crash daemons", 2024-04-16T05:47:11.814 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:47:11.814 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:47:11.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.818+0000 7f9752e00700 1 -- 172.21.15.27:0/3108455949 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f973006ec00 msgr2=0x7f97300710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:11.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.818+0000 7f9752e00700 1 --2- 172.21.15.27:0/3108455949 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f973006ec00 0x7f97300710b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f973400b020 tx=0x7f9734009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:11.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.818+0000 7f9752e00700 1 -- 172.21.15.27:0/3108455949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f974c0fe470 msgr2=0x7f974c0fda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:11.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.818+0000 7f9752e00700 1 --2- 172.21.15.27:0/3108455949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f974c0fe470 0x7f974c0fda80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f973c002a50 tx=0x7f973c0046b0 comp rx=0 tx=0).stop 2024-04-16T05:47:11.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.818+0000 7f9752e00700 1 -- 172.21.15.27:0/3108455949 shutdown_connections 2024-04-16T05:47:11.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.818+0000 7f9752e00700 1 --2- 172.21.15.27:0/3108455949 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f973006ec00 0x7f97300710b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:11.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.818+0000 7f9752e00700 1 --2- 172.21.15.27:0/3108455949 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f974c0fe470 0x7f974c0fda80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:11.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.818+0000 7f9752e00700 1 --2- 172.21.15.27:0/3108455949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f974c0fedc0 0x7f974c0fc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:11.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.818+0000 7f9752e00700 1 -- 172.21.15.27:0/3108455949 >> 172.21.15.27:0/3108455949 conn(0x7f974c0f6c10 msgr2=0x7f974c0f82c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:11.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.819+0000 7f9752e00700 1 -- 172.21.15.27:0/3108455949 shutdown_connections 2024-04-16T05:47:11.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.819+0000 7f9752e00700 1 -- 172.21.15.27:0/3108455949 wait complete. 2024-04-16T05:47:11.829 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 342, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88871 2024-04-16T05:47:11.829 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 342, success, outsize: 120 2024-04-16T05:47:11.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 344, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 88958 2024-04-16T05:47:11.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 344, success, outsize: 120 2024-04-16T05:47:11.871 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 346, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 89051 2024-04-16T05:47:11.871 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 346, success, outsize: 120 2024-04-16T05:47:11.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.964+0000 7fbde3200700 1 -- 172.21.15.27:0/902925352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbddc0fff20 msgr2=0x7fbddc102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:11.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.964+0000 7fbde3200700 1 --2- 172.21.15.27:0/902925352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbddc0fff20 0x7fbddc102340 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8004940 tx=0x7fbdc8009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:11.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.964+0000 7fbde3200700 1 -- 172.21.15.27:0/902925352 shutdown_connections 2024-04-16T05:47:11.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.964+0000 7fbde3200700 1 --2- 172.21.15.27:0/902925352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbddc0fff20 0x7fbddc102340 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:11.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.964+0000 7fbde3200700 1 --2- 172.21.15.27:0/902925352 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbddc067000 0x7fbddc0ff9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:11.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.964+0000 7fbde3200700 1 -- 172.21.15.27:0/902925352 >> 172.21.15.27:0/902925352 conn(0x7fbddc0f7450 msgr2=0x7fbddc0f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:11.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.964+0000 7fbde3200700 1 -- 172.21.15.27:0/902925352 shutdown_connections 2024-04-16T05:47:11.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.964+0000 7fbde3200700 1 -- 172.21.15.27:0/902925352 wait complete. 2024-04-16T05:47:11.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.965+0000 7fbde3200700 1 Processor -- start 2024-04-16T05:47:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.965+0000 7fbde3200700 1 -- start start 2024-04-16T05:47:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.965+0000 7fbde3200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbddc067000 0x7fbddc10faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.965+0000 7fbde3200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbddc0fff20 0x7fbddc110030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.965+0000 7fbde3200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc110630 con 0x7fbddc0fff20 2024-04-16T05:47:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.965+0000 7fbde3200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc1107a0 con 0x7fbddc067000 2024-04-16T05:47:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.965+0000 7fbddb400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbddc0fff20 0x7fbddc110030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.965+0000 7fbddb400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbddc0fff20 0x7fbddc110030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41124/0 (socket says 172.21.15.27:41124) 2024-04-16T05:47:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.965+0000 7fbddb400700 1 -- 172.21.15.27:0/3732320530 learned_addr learned my addr 172.21.15.27:0/3732320530 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:47:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.965+0000 7fbddbe00700 1 --2- 172.21.15.27:0/3732320530 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbddc067000 0x7fbddc10faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.966+0000 7fbddb400700 1 -- 172.21.15.27:0/3732320530 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbddc067000 msgr2=0x7fbddc10faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.966+0000 7fbddb400700 1 --2- 172.21.15.27:0/3732320530 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbddc067000 0x7fbddc10faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:11.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.966+0000 7fbddb400700 1 -- 172.21.15.27:0/3732320530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc8005040 con 0x7fbddc0fff20 2024-04-16T05:47:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.966+0000 7fbddb400700 1 --2- 172.21.15.27:0/3732320530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbddc0fff20 0x7fbddc110030 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8013870 tx=0x7fbdc8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.966+0000 7fbdd3e00700 1 -- 172.21.15.27:0/3732320530 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdc800e070 con 0x7fbddc0fff20 2024-04-16T05:47:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.966+0000 7fbdd3e00700 1 -- 172.21.15.27:0/3732320530 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbdc8013de0 con 0x7fbddc0fff20 2024-04-16T05:47:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.966+0000 7fbde3200700 1 -- 172.21.15.27:0/3732320530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbddc1151a0 con 0x7fbddc0fff20 2024-04-16T05:47:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.967+0000 7fbdd3e00700 1 -- 172.21.15.27:0/3732320530 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdc801cea0 con 0x7fbddc0fff20 2024-04-16T05:47:11.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.967+0000 7fbde3200700 1 -- 172.21.15.27:0/3732320530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbddc115680 con 0x7fbddc0fff20 2024-04-16T05:47:11.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.968+0000 7fbde3200700 1 -- 172.21.15.27:0/3732320530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbddc04c710 con 0x7fbddc0fff20 2024-04-16T05:47:11.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.969+0000 7fbdd3e00700 1 -- 172.21.15.27:0/3732320530 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbdc801c690 con 0x7fbddc0fff20 2024-04-16T05:47:11.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.970+0000 7fbdd3e00700 1 --2- 172.21.15.27:0/3732320530 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fbdc006ec00 0x7fbdc00710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:11.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.970+0000 7fbdd3e00700 1 -- 172.21.15.27:0/3732320530 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6496+0+0 (secure 0 0 0) 0x7fbdc808ff50 con 0x7fbddc0fff20 2024-04-16T05:47:11.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.970+0000 7fbddbe00700 1 --2- 172.21.15.27:0/3732320530 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fbdc006ec00 0x7fbdc00710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:11.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.971+0000 7fbddbe00700 1 --2- 172.21.15.27:0/3732320530 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fbdc006ec00 0x7fbdc00710b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc009b80 tx=0x7fbdcc00b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:11.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:11.975+0000 7fbdd3e00700 1 -- 172.21.15.27:0/3732320530 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fbdc805c8b0 con 0x7fbddc0fff20 2024-04-16T05:47:12.283 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:12.285+0000 7fbde3200700 1 -- 172.21.15.27:0/3732320530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fbddc105910 con 0x7fbddc0fff20 2024-04-16T05:47:12.284 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:12.286+0000 7fbdd3e00700 1 -- 172.21.15.27:0/3732320530 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fbdc8004490 con 0x7fbddc0fff20 2024-04-16T05:47:12.284 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:47:12.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:12.289+0000 7fbde3200700 1 -- 172.21.15.27:0/3732320530 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fbdc006ec00 msgr2=0x7fbdc00710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:12.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:12.289+0000 7fbde3200700 1 --2- 172.21.15.27:0/3732320530 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fbdc006ec00 0x7fbdc00710b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbdcc009b80 tx=0x7fbdcc00b040 comp rx=0 tx=0).stop 2024-04-16T05:47:12.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:12.289+0000 7fbde3200700 1 -- 172.21.15.27:0/3732320530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbddc0fff20 msgr2=0x7fbddc110030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:12.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:12.289+0000 7fbde3200700 1 --2- 172.21.15.27:0/3732320530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbddc0fff20 0x7fbddc110030 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8013870 tx=0x7fbdc8009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:12.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:12.289+0000 7fbde3200700 1 -- 172.21.15.27:0/3732320530 shutdown_connections 2024-04-16T05:47:12.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:12.289+0000 7fbde3200700 1 --2- 172.21.15.27:0/3732320530 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fbdc006ec00 0x7fbdc00710b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:12.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:12.289+0000 7fbde3200700 1 --2- 172.21.15.27:0/3732320530 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbddc067000 0x7fbddc10faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:12.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:12.289+0000 7fbde3200700 1 --2- 172.21.15.27:0/3732320530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbddc0fff20 0x7fbddc110030 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:12.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:12.289+0000 7fbde3200700 1 -- 172.21.15.27:0/3732320530 >> 172.21.15.27:0/3732320530 conn(0x7fbddc0f7450 msgr2=0x7fbddc0fba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:12.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:12.289+0000 7fbde3200700 1 -- 172.21.15.27:0/3732320530 shutdown_connections 2024-04-16T05:47:12.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:12.290+0000 7fbde3200700 1 -- 172.21.15.27:0/3732320530 wait complete. 2024-04-16T05:47:12.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:12 smithi027 ceph-mon[93635]: pgmap v54: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T05:47:12.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:12 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/3732320530' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:47:12.692 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:12 smithi122 ceph-mon[80930]: pgmap v54: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 255 B/s wr, 0 op/s 2024-04-16T05:47:12.692 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:12 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/3732320530' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:47:13.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:13 smithi122 ceph-mon[80930]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:47:13.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:13.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:13.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:13 smithi027 ceph-mon[93635]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:47:13.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:13.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:14.618 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 124, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98939 2024-04-16T05:47:14.618 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 124, success, outsize: 120 2024-04-16T05:47:14.635 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 126, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99026 2024-04-16T05:47:14.635 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 126, success, outsize: 120 2024-04-16T05:47:14.651 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 128, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99116 2024-04-16T05:47:14.651 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 128, success, outsize: 120 2024-04-16T05:47:14.668 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 130, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99212 2024-04-16T05:47:14.668 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 130, success, outsize: 120 2024-04-16T05:47:14.686 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 132, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99305 2024-04-16T05:47:14.686 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 132, success, outsize: 120 2024-04-16T05:47:14.702 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 134, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99392 2024-04-16T05:47:14.702 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 134, success, outsize: 120 2024-04-16T05:47:14.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 136, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99482 2024-04-16T05:47:14.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 136, success, outsize: 120 2024-04-16T05:47:14.736 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 138, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99578 2024-04-16T05:47:14.736 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 138, success, outsize: 120 2024-04-16T05:47:14.753 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 140, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99671 2024-04-16T05:47:14.753 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 140, success, outsize: 120 2024-04-16T05:47:14.770 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 142, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99758 2024-04-16T05:47:14.770 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 142, success, outsize: 120 2024-04-16T05:47:14.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:14 smithi122 ceph-mon[80930]: pgmap v55: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:14.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:47:14.788 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 144, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99848 2024-04-16T05:47:14.788 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 144, success, outsize: 120 2024-04-16T05:47:14.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:14 smithi027 ceph-mon[93635]: pgmap v55: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:14.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:47:14.808 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 146, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99944 2024-04-16T05:47:14.808 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 146, success, outsize: 120 2024-04-16T05:47:14.825 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 148, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100037 2024-04-16T05:47:14.825 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 148, success, outsize: 120 2024-04-16T05:47:14.843 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 150, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100124 2024-04-16T05:47:14.843 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 150, success, outsize: 120 2024-04-16T05:47:14.863 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 152, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100214 2024-04-16T05:47:14.863 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 152, success, outsize: 120 2024-04-16T05:47:14.884 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 154, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100310 2024-04-16T05:47:14.884 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 154, success, outsize: 120 2024-04-16T05:47:14.902 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 156, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100404 2024-04-16T05:47:14.902 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 156, success, outsize: 120 2024-04-16T05:47:14.918 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 158, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100491 2024-04-16T05:47:14.918 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 158, success, outsize: 120 2024-04-16T05:47:14.938 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 160, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100581 2024-04-16T05:47:14.938 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 160, success, outsize: 120 2024-04-16T05:47:14.956 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 162, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100677 2024-04-16T05:47:14.956 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 162, success, outsize: 120 2024-04-16T05:47:14.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 164, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100770 2024-04-16T05:47:14.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 164, success, outsize: 120 2024-04-16T05:47:14.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 166, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100857 2024-04-16T05:47:14.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 166, success, outsize: 120 2024-04-16T05:47:15.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 168, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100947 2024-04-16T05:47:15.002 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 168, success, outsize: 120 2024-04-16T05:47:15.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 170, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 101043 2024-04-16T05:47:15.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 170, success, outsize: 120 2024-04-16T05:47:15.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 172, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 101136 2024-04-16T05:47:15.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 172, success, outsize: 120 2024-04-16T05:47:15.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 174, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 101223 2024-04-16T05:47:15.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 174, success, outsize: 120 2024-04-16T05:47:15.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 176, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 101313 2024-04-16T05:47:15.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 176, success, outsize: 120 2024-04-16T05:47:15.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 178, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 101409 2024-04-16T05:47:15.086 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 178, success, outsize: 120 2024-04-16T05:47:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:16 smithi122 ceph-mon[80930]: pgmap v56: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:16 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:16 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:16 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:47:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:16 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:47:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:16 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:16 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:47:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:16 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:47:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:16 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:47:16.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:16 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:47:16.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:16 smithi027 ceph-mon[93635]: pgmap v56: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:16.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:16 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:16.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:16 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:16.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:16 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:47:16.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:16 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:47:16.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:16 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:16.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:16 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:47:16.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:16 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:47:16.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:16 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:47:16.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:16 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:47:17.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:17 smithi122 ceph-mon[80930]: Upgrade: Setting container_image for all crash 2024-04-16T05:47:17.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:17 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:17.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:17 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi027"}]: dispatch 2024-04-16T05:47:17.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:17 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi027"}]': finished 2024-04-16T05:47:17.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:17 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi122"}]: dispatch 2024-04-16T05:47:17.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:17 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi122"}]': finished 2024-04-16T05:47:17.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:17 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-16T05:47:17.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:17 smithi122 ceph-mon[80930]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-16T05:47:17.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:17 smithi122 ceph-mon[80930]: Upgrade: osd.0 is safe to restart 2024-04-16T05:47:17.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:17 smithi027 ceph-mon[93635]: Upgrade: Setting container_image for all crash 2024-04-16T05:47:17.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:17 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:17.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:17 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi027"}]: dispatch 2024-04-16T05:47:17.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:17 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi027"}]': finished 2024-04-16T05:47:17.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:17 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi122"}]: dispatch 2024-04-16T05:47:17.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:17 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi122"}]': finished 2024-04-16T05:47:17.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:17 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-16T05:47:17.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:17 smithi027 ceph-mon[93635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-16T05:47:17.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:17 smithi027 ceph-mon[93635]: Upgrade: osd.0 is safe to restart 2024-04-16T05:47:18.661 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:18 smithi027 ceph-mon[93635]: pgmap v57: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:19.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:18 smithi122 ceph-mon[80930]: pgmap v57: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:20.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:19 smithi027 ceph-mon[93635]: Upgrade: Updating osd.0 2024-04-16T05:47:20.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:20.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T05:47:20.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:47:20.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:19 smithi027 ceph-mon[93635]: Deploying daemon osd.0 on smithi027 2024-04-16T05:47:20.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:19 smithi027 ceph-mon[93635]: pgmap v58: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:20.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:19 smithi122 ceph-mon[80930]: Upgrade: Updating osd.0 2024-04-16T05:47:20.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:20.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-16T05:47:20.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:47:20.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:19 smithi122 ceph-mon[80930]: Deploying daemon osd.0 on smithi027 2024-04-16T05:47:20.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:19 smithi122 ceph-mon[80930]: pgmap v58: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:22.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:22 smithi122 ceph-mon[80930]: pgmap v59: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:22.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:22 smithi027 ceph-mon[93635]: pgmap v59: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:23.704 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:23 smithi027 systemd[1]: Stopping Ceph osd.0 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T05:47:24.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:24 smithi122 ceph-mon[80930]: pgmap v60: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:24.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:24 smithi027 ceph-mon[93635]: pgmap v60: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:24.307 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:24 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0[47573]: 2024-04-16T05:47:24.182+0000 7fd00f600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T05:47:24.307 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:24 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0[47573]: 2024-04-16T05:47:24.182+0000 7fd00f600700 -1 osd.0 49 *** Got signal Terminated *** 2024-04-16T05:47:24.307 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:24 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0[47573]: 2024-04-16T05:47:24.182+0000 7fd00f600700 -1 osd.0 49 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T05:47:25.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:25 smithi122 ceph-mon[80930]: osd.0 marked itself down and dead 2024-04-16T05:47:25.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:25 smithi027 ceph-mon[93635]: osd.0 marked itself down and dead 2024-04-16T05:47:25.664 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:25 smithi027 podman[101868]: 2024-04-16 05:47:25.397943635 +0000 UTC m=+1.359650744 container died 15e049573bca816175a20ef48fd8d31f7902fb072c59b17db868cad0a9168b36 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0, org.label-schema.build-date=20240409, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-16T05:47:26.022 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:25 smithi027 podman[101868]: 2024-04-16 05:47:25.842024954 +0000 UTC m=+1.803732104 container cleanup 15e049573bca816175a20ef48fd8d31f7902fb072c59b17db868cad0a9168b36 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-64502-quincy-3-45317f9) 2024-04-16T05:47:26.022 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:25 smithi027 bash[101868]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0 2024-04-16T05:47:26.307 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:26 smithi027 podman[101886]: 2024-04-16 05:47:26.034876647 +0000 UTC m=+0.623262682 container remove 15e049573bca816175a20ef48fd8d31f7902fb072c59b17db868cad0a9168b36 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T05:47:26.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:26 smithi027 ceph-mon[93635]: pgmap v61: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:26.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:26 smithi027 ceph-mon[93635]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T05:47:26.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:26 smithi027 ceph-mon[93635]: osdmap e50: 6 total, 5 up, 6 in 2024-04-16T05:47:26.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:26 smithi122 ceph-mon[80930]: pgmap v61: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:26.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:26 smithi122 ceph-mon[80930]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T05:47:26.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:26 smithi122 ceph-mon[80930]: osdmap e50: 6 total, 5 up, 6 in 2024-04-16T05:47:27.044 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:26 smithi027 podman[101975]: 2024-04-16 05:47:26.677880789 +0000 UTC m=+0.035861944 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T05:47:27.044 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:26 smithi027 podman[101975]: 2024-04-16 05:47:26.888395844 +0000 UTC m=+0.246377046 container create 9c70a8b53316ba7a18af14976f066016bf979c2c8d5105a9d032dfd7ef4bd934 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-16T05:47:27.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:27 smithi027 ceph-mon[93635]: osdmap e51: 6 total, 5 up, 6 in 2024-04-16T05:47:27.307 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:27 smithi027 podman[101975]: 2024-04-16 05:47:27.256376251 +0000 UTC m=+0.614357466 container init 9c70a8b53316ba7a18af14976f066016bf979c2c8d5105a9d032dfd7ef4bd934 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-16T05:47:27.307 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:27 smithi027 podman[101975]: 2024-04-16 05:47:27.26822437 +0000 UTC m=+0.626205515 container start 9c70a8b53316ba7a18af14976f066016bf979c2c8d5105a9d032dfd7ef4bd934 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-16T05:47:27.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:27 smithi122 ceph-mon[80930]: osdmap e51: 6 total, 5 up, 6 in 2024-04-16T05:47:27.735 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:27 smithi027 podman[101975]: 2024-04-16 05:47:27.355029978 +0000 UTC m=+0.713011180 container attach 9c70a8b53316ba7a18af14976f066016bf979c2c8d5105a9d032dfd7ef4bd934 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-16T05:47:28.047 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:27 smithi027 podman[101975]: 2024-04-16 05:47:27.738893399 +0000 UTC m=+1.096874548 container died 9c70a8b53316ba7a18af14976f066016bf979c2c8d5105a9d032dfd7ef4bd934 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-16T05:47:28.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:28 smithi027 ceph-mon[93635]: pgmap v64: 65 pgs: 9 stale+active+clean, 56 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:28.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:28 smithi122 ceph-mon[80930]: pgmap v64: 65 pgs: 9 stale+active+clean, 56 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:28.807 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:28 smithi027 podman[102001]: 2024-04-16 05:47:28.516973844 +0000 UTC m=+0.756547307 container remove 9c70a8b53316ba7a18af14976f066016bf979c2c8d5105a9d032dfd7ef4bd934 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True) 2024-04-16T05:47:28.807 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:28 smithi027 systemd[1]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.0.service: Deactivated successfully. 2024-04-16T05:47:28.807 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:28 smithi027 systemd[1]: Stopped Ceph osd.0 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968. 2024-04-16T05:47:28.807 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:28 smithi027 systemd[1]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.0.service: Consumed 13.624s CPU time. 2024-04-16T05:47:29.307 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:28 smithi027 systemd[1]: Starting Ceph osd.0 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T05:47:29.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [3, 2]}]: dispatch 2024-04-16T05:47:29.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [3, 0]}]: dispatch 2024-04-16T05:47:29.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.15", "id": [3, 2]}]: dispatch 2024-04-16T05:47:29.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [1, 5]}]: dispatch 2024-04-16T05:47:29.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [1, 2]}]: dispatch 2024-04-16T05:47:29.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [3, 0]}]: dispatch 2024-04-16T05:47:29.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [3, 2, 4, 5]}]: dispatch 2024-04-16T05:47:29.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 0]}]: dispatch 2024-04-16T05:47:29.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:29.308 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:47:29.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [3, 2]}]: dispatch 2024-04-16T05:47:29.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [3, 0]}]: dispatch 2024-04-16T05:47:29.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.15", "id": [3, 2]}]: dispatch 2024-04-16T05:47:29.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [1, 5]}]: dispatch 2024-04-16T05:47:29.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [1, 2]}]: dispatch 2024-04-16T05:47:29.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [3, 0]}]: dispatch 2024-04-16T05:47:29.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [3, 2, 4, 5]}]: dispatch 2024-04-16T05:47:29.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 0]}]: dispatch 2024-04-16T05:47:29.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:29.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:47:29.671 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:29 smithi027 podman[102103]: 2024-04-16 05:47:29.258598606 +0000 UTC m=+0.039181621 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T05:47:29.671 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:29 smithi027 podman[102103]: 2024-04-16 05:47:29.401969802 +0000 UTC m=+0.182552839 container create 035732f3dc9da42b86dcfe8c4e26805571b03cca766447457285924fbaecbd6c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-activate, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T05:47:29.671 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:29 smithi027 podman[102103]: 2024-04-16 05:47:29.672560348 +0000 UTC m=+0.453143391 container init 035732f3dc9da42b86dcfe8c4e26805571b03cca766447457285924fbaecbd6c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-16T05:47:30.057 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:29 smithi027 podman[102103]: 2024-04-16 05:47:29.685586993 +0000 UTC m=+0.466170037 container start 035732f3dc9da42b86dcfe8c4e26805571b03cca766447457285924fbaecbd6c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-activate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T05:47:30.057 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:29 smithi027 podman[102103]: 2024-04-16 05:47:29.804512101 +0000 UTC m=+0.585095188 container attach 035732f3dc9da42b86dcfe8c4e26805571b03cca766447457285924fbaecbd6c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-activate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T05:47:30.463 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 180, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102209 2024-04-16T05:47:30.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 180, success, outsize: 120 2024-04-16T05:47:30.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:30 smithi122 ceph-mon[80930]: pgmap v65: 65 pgs: 9 stale+active+clean, 56 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:30.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:30 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [3, 2]}]': finished 2024-04-16T05:47:30.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:30 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [3, 0]}]': finished 2024-04-16T05:47:30.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:30 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.15", "id": [3, 2]}]': finished 2024-04-16T05:47:30.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:30 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [1, 5]}]': finished 2024-04-16T05:47:30.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:30 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [1, 2]}]': finished 2024-04-16T05:47:30.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:30 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [3, 0]}]': finished 2024-04-16T05:47:30.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:30 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [3, 2, 4, 5]}]': finished 2024-04-16T05:47:30.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:30 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 0]}]': finished 2024-04-16T05:47:30.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:30 smithi122 ceph-mon[80930]: osdmap e52: 6 total, 5 up, 6 in 2024-04-16T05:47:30.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:30 smithi122 ceph-mon[80930]: osdmap e53: 6 total, 5 up, 6 in 2024-04-16T05:47:30.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:30 smithi027 ceph-mon[93635]: pgmap v65: 65 pgs: 9 stale+active+clean, 56 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:30.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:30 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [3, 2]}]': finished 2024-04-16T05:47:30.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:30 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [3, 0]}]': finished 2024-04-16T05:47:30.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:30 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.15", "id": [3, 2]}]': finished 2024-04-16T05:47:30.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:30 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [1, 5]}]': finished 2024-04-16T05:47:30.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:30 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [1, 2]}]': finished 2024-04-16T05:47:30.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:30 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [3, 0]}]': finished 2024-04-16T05:47:30.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:30 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [3, 2, 4, 5]}]': finished 2024-04-16T05:47:30.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:30 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 0]}]': finished 2024-04-16T05:47:30.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:30 smithi027 ceph-mon[93635]: osdmap e52: 6 total, 5 up, 6 in 2024-04-16T05:47:30.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:30 smithi027 ceph-mon[93635]: osdmap e53: 6 total, 5 up, 6 in 2024-04-16T05:47:30.962 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:30 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-activate[102117]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T05:47:30.963 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:30 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-activate[102117]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-16T05:47:30.963 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:30 smithi027 bash[102103]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T05:47:30.963 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:30 smithi027 bash[102103]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-16T05:47:30.963 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:30 smithi027 bash[102103]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-16T05:47:30.963 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:30 smithi027 bash[102103]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-16T05:47:30.963 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:30 smithi027 bash[102103]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-16T05:47:30.963 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:30 smithi027 bash[102103]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T05:47:30.963 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:30 smithi027 bash[102103]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-16T05:47:30.963 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:30 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-activate[102117]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-16T05:47:30.963 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:30 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-activate[102117]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-16T05:47:30.963 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:30 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-activate[102117]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-16T05:47:30.963 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:30 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-activate[102117]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T05:47:30.963 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:30 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-activate[102117]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-16T05:47:30.964 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:30 smithi027 podman[102247]: 2024-04-16 05:47:30.72236943 +0000 UTC m=+0.036441103 container died 035732f3dc9da42b86dcfe8c4e26805571b03cca766447457285924fbaecbd6c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-16T05:47:31.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:31 smithi027 ceph-mon[93635]: osdmap e54: 6 total, 5 up, 6 in 2024-04-16T05:47:31.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:31 smithi122 ceph-mon[80930]: osdmap e54: 6 total, 5 up, 6 in 2024-04-16T05:47:31.704 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:31 smithi027 podman[102247]: 2024-04-16 05:47:31.341821166 +0000 UTC m=+0.655892877 container remove 035732f3dc9da42b86dcfe8c4e26805571b03cca766447457285924fbaecbd6c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-activate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240409, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T05:47:32.057 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:31 smithi027 podman[102307]: 2024-04-16 05:47:31.615031702 +0000 UTC m=+0.038336117 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T05:47:32.057 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:31 smithi027 podman[102307]: 2024-04-16 05:47:31.759437023 +0000 UTC m=+0.182741468 container create cef03b67bc5ab4d7f2b8c94285b9a743e0f1da8591a2972ac73af538827948ff (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-16T05:47:32.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:32 smithi122 ceph-mon[80930]: pgmap v69: 65 pgs: 8 remapped+peering, 7 active+undersized+degraded, 22 active+undersized, 28 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T05:47:32.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:32 smithi122 ceph-mon[80930]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-16T05:47:32.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:32 smithi122 ceph-mon[80930]: Health check failed: Degraded data redundancy: 13/72 objects degraded (18.056%), 7 pgs degraded (PG_DEGRADED) 2024-04-16T05:47:32.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:32 smithi027 ceph-mon[93635]: pgmap v69: 65 pgs: 8 remapped+peering, 7 active+undersized+degraded, 22 active+undersized, 28 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T05:47:32.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:32 smithi027 ceph-mon[93635]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-16T05:47:32.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:32 smithi027 ceph-mon[93635]: Health check failed: Degraded data redundancy: 13/72 objects degraded (18.056%), 7 pgs degraded (PG_DEGRADED) 2024-04-16T05:47:32.557 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:32 smithi027 podman[102307]: 2024-04-16 05:47:32.113030459 +0000 UTC m=+0.536334927 container init cef03b67bc5ab4d7f2b8c94285b9a743e0f1da8591a2972ac73af538827948ff (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T05:47:32.557 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:32 smithi027 podman[102307]: 2024-04-16 05:47:32.126064044 +0000 UTC m=+0.549368457 container start cef03b67bc5ab4d7f2b8c94285b9a743e0f1da8591a2972ac73af538827948ff (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T05:47:32.557 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:32 smithi027 ceph-osd[102325]: -- 172.21.15.27:0/2148276674 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56125fc59680 con 0x56125fc23000 2024-04-16T05:47:32.557 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:32 smithi027 bash[102307]: cef03b67bc5ab4d7f2b8c94285b9a743e0f1da8591a2972ac73af538827948ff 2024-04-16T05:47:32.557 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:32 smithi027 systemd[1]: Started Ceph osd.0 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968. 2024-04-16T05:47:33.058 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:32 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0[102321]: 2024-04-16T05:47:32.712+0000 7f3286d047c0 -1 Falling back to public interface 2024-04-16T05:47:33.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:33 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:33.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:33 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:33.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:33 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:47:33.714 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:33 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:33.714 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:33 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:33.714 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:33 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:47:34.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:34 smithi122 ceph-mon[80930]: pgmap v70: 65 pgs: 8 remapped+peering, 7 active+undersized+degraded, 22 active+undersized, 28 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T05:47:34.795 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:34 smithi027 ceph-mon[93635]: pgmap v70: 65 pgs: 8 remapped+peering, 7 active+undersized+degraded, 22 active+undersized, 28 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T05:47:35.664 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:35 smithi027 ceph-mon[93635]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-16T05:47:35.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:35 smithi122 ceph-mon[80930]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-16T05:47:36.626 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:36 smithi027 ceph-mon[93635]: pgmap v71: 65 pgs: 8 active+clean+remapped, 7 active+undersized+degraded, 22 active+undersized, 28 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 13/72 objects degraded (18.056%); 2/72 objects misplaced (2.778%); 14 B/s, 0 objects/s recovering 2024-04-16T05:47:36.626 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:36 smithi027 ceph-mon[93635]: from='osd.0 [v2:172.21.15.27:6802/1955571017,v1:172.21.15.27:6803/1955571017]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-16T05:47:36.626 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:36 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0[102321]: 2024-04-16T05:47:36.365+0000 7f3286d047c0 -1 osd.0 49 log_to_monitors true 2024-04-16T05:47:36.626 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 05:47:36 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0[102321]: 2024-04-16T05:47:36.430+0000 7f3276000700 -1 osd.0 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T05:47:36.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:36 smithi122 ceph-mon[80930]: pgmap v71: 65 pgs: 8 active+clean+remapped, 7 active+undersized+degraded, 22 active+undersized, 28 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 13/72 objects degraded (18.056%); 2/72 objects misplaced (2.778%); 14 B/s, 0 objects/s recovering 2024-04-16T05:47:36.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:36 smithi122 ceph-mon[80930]: from='osd.0 [v2:172.21.15.27:6802/1955571017,v1:172.21.15.27:6803/1955571017]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-16T05:47:37.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:37 smithi122 ceph-mon[80930]: from='osd.0 [v2:172.21.15.27:6802/1955571017,v1:172.21.15.27:6803/1955571017]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-16T05:47:37.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:37 smithi122 ceph-mon[80930]: osdmap e55: 6 total, 5 up, 6 in 2024-04-16T05:47:37.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:37 smithi122 ceph-mon[80930]: from='osd.0 [v2:172.21.15.27:6802/1955571017,v1:172.21.15.27:6803/1955571017]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-16T05:47:37.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:37 smithi027 ceph-mon[93635]: from='osd.0 [v2:172.21.15.27:6802/1955571017,v1:172.21.15.27:6803/1955571017]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-16T05:47:37.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:37 smithi027 ceph-mon[93635]: osdmap e55: 6 total, 5 up, 6 in 2024-04-16T05:47:37.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:37 smithi027 ceph-mon[93635]: from='osd.0 [v2:172.21.15.27:6802/1955571017,v1:172.21.15.27:6803/1955571017]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-16T05:47:38.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:38 smithi122 ceph-mon[80930]: pgmap v73: 65 pgs: 8 active+clean+remapped, 7 active+undersized+degraded, 22 active+undersized, 28 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 13/72 objects degraded (18.056%); 2/72 objects misplaced (2.778%); 11 B/s, 0 objects/s recovering 2024-04-16T05:47:38.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:38 smithi122 ceph-mon[80930]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T05:47:38.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:38 smithi122 ceph-mon[80930]: osd.0 [v2:172.21.15.27:6802/1955571017,v1:172.21.15.27:6803/1955571017] boot 2024-04-16T05:47:38.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:38 smithi122 ceph-mon[80930]: osdmap e56: 6 total, 6 up, 6 in 2024-04-16T05:47:38.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:38 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T05:47:38.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:38 smithi122 ceph-mon[80930]: osdmap e57: 6 total, 6 up, 6 in 2024-04-16T05:47:38.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:38 smithi027 ceph-mon[93635]: pgmap v73: 65 pgs: 8 active+clean+remapped, 7 active+undersized+degraded, 22 active+undersized, 28 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 13/72 objects degraded (18.056%); 2/72 objects misplaced (2.778%); 11 B/s, 0 objects/s recovering 2024-04-16T05:47:38.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:38 smithi027 ceph-mon[93635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T05:47:38.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:38 smithi027 ceph-mon[93635]: osd.0 [v2:172.21.15.27:6802/1955571017,v1:172.21.15.27:6803/1955571017] boot 2024-04-16T05:47:38.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:38 smithi027 ceph-mon[93635]: osdmap e56: 6 total, 6 up, 6 in 2024-04-16T05:47:38.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:38 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-16T05:47:38.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:38 smithi027 ceph-mon[93635]: osdmap e57: 6 total, 6 up, 6 in 2024-04-16T05:47:40.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:40 smithi122 ceph-mon[80930]: pgmap v76: 65 pgs: 8 active+clean+remapped, 7 active+undersized+degraded, 22 active+undersized, 28 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 13/72 objects degraded (18.056%); 2/72 objects misplaced (2.778%); 14 B/s, 0 objects/s recovering 2024-04-16T05:47:40.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:40 smithi122 ceph-mon[80930]: osdmap e58: 6 total, 6 up, 6 in 2024-04-16T05:47:40.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:40 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:40.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:40 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:40 smithi027 ceph-mon[93635]: pgmap v76: 65 pgs: 8 active+clean+remapped, 7 active+undersized+degraded, 22 active+undersized, 28 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 13/72 objects degraded (18.056%); 2/72 objects misplaced (2.778%); 14 B/s, 0 objects/s recovering 2024-04-16T05:47:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:40 smithi027 ceph-mon[93635]: osdmap e58: 6 total, 6 up, 6 in 2024-04-16T05:47:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:41.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:41 smithi122 ceph-mon[80930]: Health check update: Degraded data redundancy: 4/72 objects degraded (5.556%), 2 pgs degraded (PG_DEGRADED) 2024-04-16T05:47:41.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:41 smithi027 ceph-mon[93635]: Health check update: Degraded data redundancy: 4/72 objects degraded (5.556%), 2 pgs degraded (PG_DEGRADED) 2024-04-16T05:47:42.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.459+0000 7f2224e00700 1 -- 172.21.15.27:0/2081696447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22201002b0 msgr2=0x7f2220100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:42.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.459+0000 7f2224e00700 1 --2- 172.21.15.27:0/2081696447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22201002b0 0x7f2220100720 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f220c004940 tx=0x7f220c009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:42.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.459+0000 7f2224e00700 1 -- 172.21.15.27:0/2081696447 shutdown_connections 2024-04-16T05:47:42.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.459+0000 7f2224e00700 1 --2- 172.21.15.27:0/2081696447 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22201002b0 0x7f2220100720 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:42.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.459+0000 7f2224e00700 1 --2- 172.21.15.27:0/2081696447 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22200ff000 0x7f22200ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:42.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.459+0000 7f2224e00700 1 -- 172.21.15.27:0/2081696447 >> 172.21.15.27:0/2081696447 conn(0x7f22200fa950 msgr2=0x7f22200fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:42.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.459+0000 7f2224e00700 1 -- 172.21.15.27:0/2081696447 shutdown_connections 2024-04-16T05:47:42.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.459+0000 7f2224e00700 1 -- 172.21.15.27:0/2081696447 wait complete. 2024-04-16T05:47:42.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.460+0000 7f2224e00700 1 Processor -- start 2024-04-16T05:47:42.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.460+0000 7f2224e00700 1 -- start start 2024-04-16T05:47:42.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.460+0000 7f2224e00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22200ff000 0x7f222010fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:42.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.460+0000 7f2224e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22201002b0 0x7f22201101d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:42.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.460+0000 7f2224e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22201107d0 con 0x7f22201002b0 2024-04-16T05:47:42.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.460+0000 7f2224e00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2220110fd0 con 0x7f22200ff000 2024-04-16T05:47:42.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.461+0000 7f221d400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22201002b0 0x7f22201101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:42.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.461+0000 7f221d400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22201002b0 0x7f22201101d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53806/0 (socket says 172.21.15.27:53806) 2024-04-16T05:47:42.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.461+0000 7f221d400700 1 -- 172.21.15.27:0/2750658001 learned_addr learned my addr 172.21.15.27:0/2750658001 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:47:42.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.461+0000 7f221de00700 1 --2- 172.21.15.27:0/2750658001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22200ff000 0x7f222010fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:42.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.461+0000 7f221d400700 1 -- 172.21.15.27:0/2750658001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22200ff000 msgr2=0x7f222010fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:42.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.461+0000 7f221d400700 1 --2- 172.21.15.27:0/2750658001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22200ff000 0x7f222010fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:42.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.461+0000 7f221d400700 1 -- 172.21.15.27:0/2750658001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220c005040 con 0x7f22201002b0 2024-04-16T05:47:42.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.461+0000 7f221de00700 1 --2- 172.21.15.27:0/2750658001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22200ff000 0x7f222010fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:47:42.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.462+0000 7f221d400700 1 --2- 172.21.15.27:0/2750658001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22201002b0 0x7f22201101d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f220c009150 tx=0x7f220c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:42.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.462+0000 7f2216000700 1 -- 172.21.15.27:0/2750658001 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f220c00e040 con 0x7f22201002b0 2024-04-16T05:47:42.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.462+0000 7f2216000700 1 -- 172.21.15.27:0/2750658001 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f220c00bcc0 con 0x7f22201002b0 2024-04-16T05:47:42.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.462+0000 7f2216000700 1 -- 172.21.15.27:0/2750658001 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f220c01c630 con 0x7f22201002b0 2024-04-16T05:47:42.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.462+0000 7f2224e00700 1 -- 172.21.15.27:0/2750658001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2220111250 con 0x7f22201002b0 2024-04-16T05:47:42.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.462+0000 7f2224e00700 1 -- 172.21.15.27:0/2750658001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2220111790 con 0x7f22201002b0 2024-04-16T05:47:42.462 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.464+0000 7f2224e00700 1 -- 172.21.15.27:0/2750658001 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f222005f5f0 con 0x7f22201002b0 2024-04-16T05:47:42.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.465+0000 7f2216000700 1 -- 172.21.15.27:0/2750658001 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f220c013420 con 0x7f22201002b0 2024-04-16T05:47:42.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.466+0000 7f2216000700 1 --2- 172.21.15.27:0/2750658001 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f220406ec00 0x7f22040710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:42.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.466+0000 7f2216000700 1 -- 172.21.15.27:0/2750658001 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f220c08f520 con 0x7f22201002b0 2024-04-16T05:47:42.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.467+0000 7f221de00700 1 --2- 172.21.15.27:0/2750658001 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f220406ec00 0x7f22040710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:42.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.467+0000 7f221de00700 1 --2- 172.21.15.27:0/2750658001 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f220406ec00 0x7f22040710b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2210007fd0 tx=0x7f2210009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:42.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.472+0000 7f2216000700 1 -- 172.21.15.27:0/2750658001 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f220c05be50 con 0x7f22201002b0 2024-04-16T05:47:42.708 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:42 smithi027 ceph-mon[93635]: pgmap v78: 65 pgs: 2 active+clean+remapped, 2 active+undersized+degraded, 6 active+undersized, 55 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 4/72 objects degraded (5.556%) 2024-04-16T05:47:42.708 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:42 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:42.708 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:42 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:42.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.739+0000 7f2224e00700 1 -- 172.21.15.27:0/2750658001 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2220111ae0 con 0x7f220406ec00 2024-04-16T05:47:42.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.740+0000 7f2216000700 1 -- 172.21.15.27:0/2750658001 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f2220111ae0 con 0x7f220406ec00 2024-04-16T05:47:42.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.745+0000 7f2224e00700 1 -- 172.21.15.27:0/2750658001 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f220406ec00 msgr2=0x7f22040710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:42.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.745+0000 7f2224e00700 1 --2- 172.21.15.27:0/2750658001 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f220406ec00 0x7f22040710b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2210007fd0 tx=0x7f2210009040 comp rx=0 tx=0).stop 2024-04-16T05:47:42.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.745+0000 7f2224e00700 1 -- 172.21.15.27:0/2750658001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22201002b0 msgr2=0x7f22201101d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:42.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.745+0000 7f2224e00700 1 --2- 172.21.15.27:0/2750658001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22201002b0 0x7f22201101d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f220c009150 tx=0x7f220c009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:42.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.745+0000 7f2224e00700 1 -- 172.21.15.27:0/2750658001 shutdown_connections 2024-04-16T05:47:42.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.745+0000 7f2224e00700 1 --2- 172.21.15.27:0/2750658001 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f220406ec00 0x7f22040710b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:42.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.745+0000 7f2224e00700 1 --2- 172.21.15.27:0/2750658001 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22200ff000 0x7f222010fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:42.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.745+0000 7f2224e00700 1 --2- 172.21.15.27:0/2750658001 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22201002b0 0x7f22201101d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:42.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.745+0000 7f2224e00700 1 -- 172.21.15.27:0/2750658001 >> 172.21.15.27:0/2750658001 conn(0x7f22200fa950 msgr2=0x7f22200fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:42.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.746+0000 7f2224e00700 1 -- 172.21.15.27:0/2750658001 shutdown_connections 2024-04-16T05:47:42.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.746+0000 7f2224e00700 1 -- 172.21.15.27:0/2750658001 wait complete. 2024-04-16T05:47:42.770 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:47:42.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:42 smithi122 ceph-mon[80930]: pgmap v78: 65 pgs: 2 active+clean+remapped, 2 active+undersized+degraded, 6 active+undersized, 55 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 4/72 objects degraded (5.556%) 2024-04-16T05:47:42.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:42 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:42.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:42 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:42.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.923+0000 7ff14a000700 1 -- 172.21.15.27:0/844219763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1440ff020 msgr2=0x7ff1440ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:42.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.923+0000 7ff14a000700 1 --2- 172.21.15.27:0/844219763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1440ff020 0x7ff1440ff430 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff134004940 tx=0x7ff134009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:42.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.923+0000 7ff14a000700 1 -- 172.21.15.27:0/844219763 shutdown_connections 2024-04-16T05:47:42.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.923+0000 7ff14a000700 1 --2- 172.21.15.27:0/844219763 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1441002d0 0x7ff144100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:42.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.923+0000 7ff14a000700 1 --2- 172.21.15.27:0/844219763 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1440ff020 0x7ff1440ff430 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:42.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.923+0000 7ff14a000700 1 -- 172.21.15.27:0/844219763 >> 172.21.15.27:0/844219763 conn(0x7ff1440fa950 msgr2=0x7ff1440fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:42.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.923+0000 7ff14a000700 1 -- 172.21.15.27:0/844219763 shutdown_connections 2024-04-16T05:47:42.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.923+0000 7ff14a000700 1 -- 172.21.15.27:0/844219763 wait complete. 2024-04-16T05:47:42.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.924+0000 7ff14a000700 1 Processor -- start 2024-04-16T05:47:42.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.924+0000 7ff14a000700 1 -- start start 2024-04-16T05:47:42.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.925+0000 7ff14a000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1440ff020 0x7ff14410fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:42.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.925+0000 7ff14a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1441002d0 0x7ff144110150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:42.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.925+0000 7ff14a000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff144110750 con 0x7ff1441002d0 2024-04-16T05:47:42.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.925+0000 7ff14a000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff144110fa0 con 0x7ff1440ff020 2024-04-16T05:47:42.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.925+0000 7ff142a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1441002d0 0x7ff144110150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:42.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.925+0000 7ff142a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1441002d0 0x7ff144110150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53824/0 (socket says 172.21.15.27:53824) 2024-04-16T05:47:42.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.925+0000 7ff142a00700 1 -- 172.21.15.27:0/1531952671 learned_addr learned my addr 172.21.15.27:0/1531952671 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:47:42.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.926+0000 7ff143400700 1 --2- 172.21.15.27:0/1531952671 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1440ff020 0x7ff14410fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:42.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.926+0000 7ff142a00700 1 -- 172.21.15.27:0/1531952671 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1440ff020 msgr2=0x7ff14410fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:42.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.926+0000 7ff142a00700 1 --2- 172.21.15.27:0/1531952671 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1440ff020 0x7ff14410fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:42.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.926+0000 7ff142a00700 1 -- 172.21.15.27:0/1531952671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff134005040 con 0x7ff1441002d0 2024-04-16T05:47:42.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.926+0000 7ff142a00700 1 --2- 172.21.15.27:0/1531952671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1441002d0 0x7ff144110150 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff130002a50 tx=0x7ff1300046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:42.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.926+0000 7ff13b400700 1 -- 172.21.15.27:0/1531952671 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff130007cc0 con 0x7ff1441002d0 2024-04-16T05:47:42.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.927+0000 7ff13b400700 1 -- 172.21.15.27:0/1531952671 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff13000f040 con 0x7ff1441002d0 2024-04-16T05:47:42.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.927+0000 7ff13b400700 1 -- 172.21.15.27:0/1531952671 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff130009640 con 0x7ff1441002d0 2024-04-16T05:47:42.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.927+0000 7ff14a000700 1 -- 172.21.15.27:0/1531952671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff144111280 con 0x7ff1441002d0 2024-04-16T05:47:42.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.927+0000 7ff14a000700 1 -- 172.21.15.27:0/1531952671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1441117c0 con 0x7ff1441002d0 2024-04-16T05:47:42.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.928+0000 7ff14a000700 1 -- 172.21.15.27:0/1531952671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14405f5f0 con 0x7ff1441002d0 2024-04-16T05:47:42.928 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.929+0000 7ff13b400700 1 -- 172.21.15.27:0/1531952671 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff1300097c0 con 0x7ff1441002d0 2024-04-16T05:47:42.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.930+0000 7ff13b400700 1 --2- 172.21.15.27:0/1531952671 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff12806ec10 0x7ff1280710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:42.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.930+0000 7ff13b400700 1 -- 172.21.15.27:0/1531952671 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6736+0+0 (secure 0 0 0) 0x7ff13008e640 con 0x7ff1441002d0 2024-04-16T05:47:42.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.931+0000 7ff143400700 1 --2- 172.21.15.27:0/1531952671 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff12806ec10 0x7ff1280710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:42.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.931+0000 7ff143400700 1 --2- 172.21.15.27:0/1531952671 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff12806ec10 0x7ff1280710c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff134004f90 tx=0x7ff134009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:42.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:42.936+0000 7ff13b400700 1 -- 172.21.15.27:0/1531952671 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff13005aeb0 con 0x7ff1441002d0 2024-04-16T05:47:43.200 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.201+0000 7ff14a000700 1 -- 172.21.15.27:0/1531952671 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff144111af0 con 0x7ff12806ec10 2024-04-16T05:47:43.201 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.202+0000 7ff13b400700 1 -- 172.21.15.27:0/1531952671 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7ff144111af0 con 0x7ff12806ec10 2024-04-16T05:47:43.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.205+0000 7ff14a000700 1 -- 172.21.15.27:0/1531952671 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff12806ec10 msgr2=0x7ff1280710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:43.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.205+0000 7ff14a000700 1 --2- 172.21.15.27:0/1531952671 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff12806ec10 0x7ff1280710c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff134004f90 tx=0x7ff134009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:43.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.205+0000 7ff14a000700 1 -- 172.21.15.27:0/1531952671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1441002d0 msgr2=0x7ff144110150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:43.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.205+0000 7ff14a000700 1 --2- 172.21.15.27:0/1531952671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1441002d0 0x7ff144110150 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff130002a50 tx=0x7ff1300046b0 comp rx=0 tx=0).stop 2024-04-16T05:47:43.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.206+0000 7ff14a000700 1 -- 172.21.15.27:0/1531952671 shutdown_connections 2024-04-16T05:47:43.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.206+0000 7ff14a000700 1 --2- 172.21.15.27:0/1531952671 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff12806ec10 0x7ff1280710c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:43.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.206+0000 7ff14a000700 1 --2- 172.21.15.27:0/1531952671 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1440ff020 0x7ff14410fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:43.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.206+0000 7ff14a000700 1 --2- 172.21.15.27:0/1531952671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1441002d0 0x7ff144110150 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:43.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.206+0000 7ff14a000700 1 -- 172.21.15.27:0/1531952671 >> 172.21.15.27:0/1531952671 conn(0x7ff1440fa950 msgr2=0x7ff1440fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:43.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.206+0000 7ff14a000700 1 -- 172.21.15.27:0/1531952671 shutdown_connections 2024-04-16T05:47:43.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.206+0000 7ff14a000700 1 -- 172.21.15.27:0/1531952671 wait complete. 2024-04-16T05:47:43.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.361+0000 7fe6d3e00700 1 -- 172.21.15.27:0/1221117575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6cc0fe040 msgr2=0x7fe6cc0fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:43.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.361+0000 7fe6d3e00700 1 --2- 172.21.15.27:0/1221117575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6cc0fe040 0x7fe6cc0fe4b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8004940 tx=0x7fe6b8009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:43.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.361+0000 7fe6d3e00700 1 -- 172.21.15.27:0/1221117575 shutdown_connections 2024-04-16T05:47:43.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.361+0000 7fe6d3e00700 1 --2- 172.21.15.27:0/1221117575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6cc0fe040 0x7fe6cc0fe4b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:43.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.361+0000 7fe6d3e00700 1 --2- 172.21.15.27:0/1221117575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6cc0fce40 0x7fe6cc0fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:43.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.361+0000 7fe6d3e00700 1 -- 172.21.15.27:0/1221117575 >> 172.21.15.27:0/1221117575 conn(0x7fe6cc0f8530 msgr2=0x7fe6cc0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:43.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.362+0000 7fe6d3e00700 1 -- 172.21.15.27:0/1221117575 shutdown_connections 2024-04-16T05:47:43.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.362+0000 7fe6d3e00700 1 -- 172.21.15.27:0/1221117575 wait complete. 2024-04-16T05:47:43.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.363+0000 7fe6d3e00700 1 Processor -- start 2024-04-16T05:47:43.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.363+0000 7fe6d3e00700 1 -- start start 2024-04-16T05:47:43.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.363+0000 7fe6d3e00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6cc0fce40 0x7fe6cc1070c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:43.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.363+0000 7fe6d3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6cc0fe040 0x7fe6cc107600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:43.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.363+0000 7fe6d3e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc107c50 con 0x7fe6cc0fe040 2024-04-16T05:47:43.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.363+0000 7fe6d3e00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6cc1084a0 con 0x7fe6cc0fce40 2024-04-16T05:47:43.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.364+0000 7fe6d0a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6cc0fe040 0x7fe6cc107600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:43.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.364+0000 7fe6d0a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6cc0fe040 0x7fe6cc107600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53828/0 (socket says 172.21.15.27:53828) 2024-04-16T05:47:43.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.364+0000 7fe6d0a00700 1 -- 172.21.15.27:0/1971102317 learned_addr learned my addr 172.21.15.27:0/1971102317 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:47:43.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.364+0000 7fe6d1400700 1 --2- 172.21.15.27:0/1971102317 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6cc0fce40 0x7fe6cc1070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:43.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.364+0000 7fe6d0a00700 1 -- 172.21.15.27:0/1971102317 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6cc0fce40 msgr2=0x7fe6cc1070c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:43.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.364+0000 7fe6d0a00700 1 --2- 172.21.15.27:0/1971102317 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6cc0fce40 0x7fe6cc1070c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:43.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.364+0000 7fe6d0a00700 1 -- 172.21.15.27:0/1971102317 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b8005040 con 0x7fe6cc0fe040 2024-04-16T05:47:43.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.364+0000 7fe6d1400700 1 --2- 172.21.15.27:0/1971102317 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6cc0fce40 0x7fe6cc1070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T05:47:43.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.365+0000 7fe6d0a00700 1 --2- 172.21.15.27:0/1971102317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6cc0fe040 0x7fe6cc107600 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe6b800b3b0 tx=0x7fe6b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:43.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.365+0000 7fe6c9600700 1 -- 172.21.15.27:0/1971102317 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6b800e040 con 0x7fe6cc0fe040 2024-04-16T05:47:43.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.365+0000 7fe6c9600700 1 -- 172.21.15.27:0/1971102317 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe6b8004290 con 0x7fe6cc0fe040 2024-04-16T05:47:43.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.365+0000 7fe6d3e00700 1 -- 172.21.15.27:0/1971102317 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6cc108720 con 0x7fe6cc0fe040 2024-04-16T05:47:43.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.365+0000 7fe6c9600700 1 -- 172.21.15.27:0/1971102317 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6b801c630 con 0x7fe6cc0fe040 2024-04-16T05:47:43.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.365+0000 7fe6d3e00700 1 -- 172.21.15.27:0/1971102317 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6cc108ce0 con 0x7fe6cc0fe040 2024-04-16T05:47:43.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.366+0000 7fe6d3e00700 1 -- 172.21.15.27:0/1971102317 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6cc1019d0 con 0x7fe6cc0fe040 2024-04-16T05:47:43.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.369+0000 7fe6c9600700 1 -- 172.21.15.27:0/1971102317 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe6b800bbf0 con 0x7fe6cc0fe040 2024-04-16T05:47:43.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.370+0000 7fe6c9600700 1 --2- 172.21.15.27:0/1971102317 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe6b406ec10 0x7fe6b40710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:43.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.370+0000 7fe6c9600700 1 -- 172.21.15.27:0/1971102317 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6736+0+0 (secure 0 0 0) 0x7fe6b8090810 con 0x7fe6cc0fe040 2024-04-16T05:47:43.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.370+0000 7fe6d1400700 1 --2- 172.21.15.27:0/1971102317 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe6b406ec10 0x7fe6b40710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:43.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.371+0000 7fe6d1400700 1 --2- 172.21.15.27:0/1971102317 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe6b406ec10 0x7fe6b40710c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0009fd0 tx=0x7fe6c0009f00 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:43.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.374+0000 7fe6c9600700 1 -- 172.21.15.27:0/1971102317 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe6b8059610 con 0x7fe6cc0fe040 2024-04-16T05:47:43.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.626+0000 7fe6d3e00700 1 -- 172.21.15.27:0/1971102317 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6cc105a40 con 0x7fe6b406ec10 2024-04-16T05:47:43.625 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:43 smithi027 ceph-mon[93635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/72 objects degraded (5.556%), 2 pgs degraded) 2024-04-16T05:47:43.625 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:43 smithi027 ceph-mon[93635]: Cluster is now healthy 2024-04-16T05:47:43.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.647+0000 7fe6c9600700 1 -- 172.21.15.27:0/1971102317 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fe6cc105a40 con 0x7fe6b406ec10 2024-04-16T05:47:43.647 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:47:43.647 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (4m) 3s ago 14m 13.6M - 0.25.0 c8568f914cd2 4ed63a8a6b33 2024-04-16T05:47:43.647 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (50s) 3s ago 15m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 4aa1f0006685 2024-04-16T05:47:43.647 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (43s) 38s ago 13m 6656k - 19.0.0-2884-g218a6914 a4e4811815e4 1c1912507d56 2024-04-16T05:47:43.647 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (3m) 3s ago 14m 75.4M - 9.4.7 954c08fa6188 f7ee2a1afd12 2024-04-16T05:47:43.647 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (7m) 3s ago 7m 23.7M - 17.2.7-847-g45317f95 b333581ccdf5 0824e2621be5 2024-04-16T05:47:43.647 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (7m) 3s ago 7m 13.7M - 17.2.7-847-g45317f95 b333581ccdf5 b332a72cc594 2024-04-16T05:47:43.647 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (7m) 38s ago 7m 13.6M - 17.2.7-847-g45317f95 b333581ccdf5 f5e36815ddfa 2024-04-16T05:47:43.647 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (7m) 38s ago 7m 13.5M - 17.2.7-847-g45317f95 b333581ccdf5 755a73c65a7f 2024-04-16T05:47:43.647 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:8443,9283,8765 running (3m) 3s ago 16m 499M - 19.0.0-2884-g218a6914 a4e4811815e4 b69a0b08e12b 2024-04-16T05:47:43.647 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283,8765 running (2m) 38s ago 13m 451M - 19.0.0-2884-g218a6914 a4e4811815e4 32e27bcd7e42 2024-04-16T05:47:43.647 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (2m) 3s ago 16m 42.6M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 299ca337e4ca 2024-04-16T05:47:43.647 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (112s) 38s ago 13m 31.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5db5f3037ed0 2024-04-16T05:47:43.648 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (4m) 3s ago 14m 7826k - 1.5.0 0da6a335fe13 5e29e29868b0 2024-04-16T05:47:43.648 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (3m) 38s ago 13m 8032k - 1.5.0 0da6a335fe13 a360922bc621 2024-04-16T05:47:43.648 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (12s) 3s ago 12m 11.2M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 cef03b67bc5a 2024-04-16T05:47:43.648 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (11m) 3s ago 11m 47.9M 1557M 17.2.7-847-g45317f95 b333581ccdf5 e146295f6500 2024-04-16T05:47:43.648 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (11m) 3s ago 10m 50.2M 1557M 17.2.7-847-g45317f95 b333581ccdf5 6212f6a18fa4 2024-04-16T05:47:43.648 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (10m) 38s ago 10m 46.3M 2581M 17.2.7-847-g45317f95 b333581ccdf5 d6724ad05f9f 2024-04-16T05:47:43.648 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (9m) 38s ago 9m 45.0M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7a59908a3e10 2024-04-16T05:47:43.648 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (9m) 38s ago 9m 44.2M 2581M 17.2.7-847-g45317f95 b333581ccdf5 8e4b60163907 2024-04-16T05:47:43.648 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (2m) 3s ago 14m 54.8M - 2.43.0 a07b618ecd1d e70f55ba5ec0 2024-04-16T05:47:43.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.650+0000 7fe6d3e00700 1 -- 172.21.15.27:0/1971102317 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe6b406ec10 msgr2=0x7fe6b40710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:43.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.650+0000 7fe6d3e00700 1 --2- 172.21.15.27:0/1971102317 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe6b406ec10 0x7fe6b40710c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0009fd0 tx=0x7fe6c0009f00 comp rx=0 tx=0).stop 2024-04-16T05:47:43.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.650+0000 7fe6d3e00700 1 -- 172.21.15.27:0/1971102317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6cc0fe040 msgr2=0x7fe6cc107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:43.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.650+0000 7fe6d3e00700 1 --2- 172.21.15.27:0/1971102317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6cc0fe040 0x7fe6cc107600 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe6b800b3b0 tx=0x7fe6b8009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:43.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.650+0000 7fe6d3e00700 1 -- 172.21.15.27:0/1971102317 shutdown_connections 2024-04-16T05:47:43.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.650+0000 7fe6d3e00700 1 --2- 172.21.15.27:0/1971102317 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe6b406ec10 0x7fe6b40710c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:43.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.650+0000 7fe6d3e00700 1 --2- 172.21.15.27:0/1971102317 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe6cc0fce40 0x7fe6cc1070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:43.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.650+0000 7fe6d3e00700 1 --2- 172.21.15.27:0/1971102317 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe6cc0fe040 0x7fe6cc107600 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:43.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.650+0000 7fe6d3e00700 1 -- 172.21.15.27:0/1971102317 >> 172.21.15.27:0/1971102317 conn(0x7fe6cc0f8530 msgr2=0x7fe6cc0f9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:43.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.650+0000 7fe6d3e00700 1 -- 172.21.15.27:0/1971102317 shutdown_connections 2024-04-16T05:47:43.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.650+0000 7fe6d3e00700 1 -- 172.21.15.27:0/1971102317 wait complete. 2024-04-16T05:47:43.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.798+0000 7fc5a1200700 1 -- 172.21.15.27:0/3694473998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59c1046d0 msgr2=0x7fc59c104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:43.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.798+0000 7fc5a1200700 1 --2- 172.21.15.27:0/3694473998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59c1046d0 0x7fc59c104b40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc584004940 tx=0x7fc584009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:43.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.798+0000 7fc5a1200700 1 -- 172.21.15.27:0/3694473998 shutdown_connections 2024-04-16T05:47:43.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.798+0000 7fc5a1200700 1 --2- 172.21.15.27:0/3694473998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59c1046d0 0x7fc59c104b40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:43.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.798+0000 7fc5a1200700 1 --2- 172.21.15.27:0/3694473998 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc59c103420 0x7fc59c103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:43.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.798+0000 7fc5a1200700 1 -- 172.21.15.27:0/3694473998 >> 172.21.15.27:0/3694473998 conn(0x7fc59c072640 msgr2=0x7fc59c074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:43.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.798+0000 7fc5a1200700 1 -- 172.21.15.27:0/3694473998 shutdown_connections 2024-04-16T05:47:43.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.798+0000 7fc5a1200700 1 -- 172.21.15.27:0/3694473998 wait complete. 2024-04-16T05:47:43.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.799+0000 7fc5a1200700 1 Processor -- start 2024-04-16T05:47:43.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.799+0000 7fc5a1200700 1 -- start start 2024-04-16T05:47:43.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.799+0000 7fc5a1200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc59c103420 0x7fc59c113f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:43.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.799+0000 7fc5a1200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59c1046d0 0x7fc59c114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:43.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.799+0000 7fc5a1200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c114a90 con 0x7fc59c1046d0 2024-04-16T05:47:43.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.799+0000 7fc5a1200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c114c00 con 0x7fc59c103420 2024-04-16T05:47:43.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.800+0000 7fc591400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59c1046d0 0x7fc59c114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:43.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.800+0000 7fc591400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59c1046d0 0x7fc59c114490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53836/0 (socket says 172.21.15.27:53836) 2024-04-16T05:47:43.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.800+0000 7fc591400700 1 -- 172.21.15.27:0/619128135 learned_addr learned my addr 172.21.15.27:0/619128135 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:47:43.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.800+0000 7fc599e00700 1 --2- 172.21.15.27:0/619128135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc59c103420 0x7fc59c113f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:43.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.800+0000 7fc591400700 1 -- 172.21.15.27:0/619128135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc59c103420 msgr2=0x7fc59c113f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:43.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.800+0000 7fc591400700 1 --2- 172.21.15.27:0/619128135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc59c103420 0x7fc59c113f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:43.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.800+0000 7fc591400700 1 -- 172.21.15.27:0/619128135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc584005040 con 0x7fc59c1046d0 2024-04-16T05:47:43.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.800+0000 7fc599e00700 1 --2- 172.21.15.27:0/619128135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc59c103420 0x7fc59c113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:47:43.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.801+0000 7fc591400700 1 --2- 172.21.15.27:0/619128135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59c1046d0 0x7fc59c114490 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc58400b3b0 tx=0x7fc584009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:43.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.801+0000 7fc592a00700 1 -- 172.21.15.27:0/619128135 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc58400e040 con 0x7fc59c1046d0 2024-04-16T05:47:43.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.801+0000 7fc592a00700 1 -- 172.21.15.27:0/619128135 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc584004290 con 0x7fc59c1046d0 2024-04-16T05:47:43.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.801+0000 7fc5a1200700 1 -- 172.21.15.27:0/619128135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc59c119600 con 0x7fc59c1046d0 2024-04-16T05:47:43.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.801+0000 7fc592a00700 1 -- 172.21.15.27:0/619128135 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc58401c630 con 0x7fc59c1046d0 2024-04-16T05:47:43.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.801+0000 7fc5a1200700 1 -- 172.21.15.27:0/619128135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc59c119bc0 con 0x7fc59c1046d0 2024-04-16T05:47:43.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.803+0000 7fc5a1200700 1 -- 172.21.15.27:0/619128135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5780052e0 con 0x7fc59c1046d0 2024-04-16T05:47:43.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.803+0000 7fc592a00700 1 -- 172.21.15.27:0/619128135 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc584013420 con 0x7fc59c1046d0 2024-04-16T05:47:43.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.804+0000 7fc592a00700 1 --2- 172.21.15.27:0/619128135 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc58006ec00 0x7fc5800710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:43.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.804+0000 7fc592a00700 1 -- 172.21.15.27:0/619128135 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6736+0+0 (secure 0 0 0) 0x7fc58408fce0 con 0x7fc59c1046d0 2024-04-16T05:47:43.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.805+0000 7fc599e00700 1 --2- 172.21.15.27:0/619128135 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc58006ec00 0x7fc5800710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:43.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.805+0000 7fc599e00700 1 --2- 172.21.15.27:0/619128135 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc58006ec00 0x7fc5800710b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc58c007fd0 tx=0x7fc58c009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:43.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:43.809+0000 7fc592a00700 1 -- 172.21.15.27:0/619128135 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc58405c550 con 0x7fc59c1046d0 2024-04-16T05:47:44.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:43 smithi122 ceph-mon[80930]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/72 objects degraded (5.556%), 2 pgs degraded) 2024-04-16T05:47:44.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:43 smithi122 ceph-mon[80930]: Cluster is now healthy 2024-04-16T05:47:44.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.144+0000 7fc5a1200700 1 -- 172.21.15.27:0/619128135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fc578005da0 con 0x7fc59c1046d0 2024-04-16T05:47:44.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.145+0000 7fc592a00700 1 -- 172.21.15.27:0/619128135 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7fc58405c0e0 con 0x7fc59c1046d0 2024-04-16T05:47:44.144 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:47:44.145 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:47:44.145 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:47:44.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:47:44.145 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:47:44.145 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:47:44.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:47:44.145 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:47:44.145 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 5, 2024-04-16T05:47:44.145 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 1 2024-04-16T05:47:44.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:47:44.145 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:47:44.145 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T05:47:44.145 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:47:44.145 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:47:44.145 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 9, 2024-04-16T05:47:44.146 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 5 2024-04-16T05:47:44.146 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:47:44.146 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:47:44.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.148+0000 7fc5a1200700 1 -- 172.21.15.27:0/619128135 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc58006ec00 msgr2=0x7fc5800710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:44.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.148+0000 7fc5a1200700 1 --2- 172.21.15.27:0/619128135 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc58006ec00 0x7fc5800710b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc58c007fd0 tx=0x7fc58c009040 comp rx=0 tx=0).stop 2024-04-16T05:47:44.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.148+0000 7fc5a1200700 1 -- 172.21.15.27:0/619128135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59c1046d0 msgr2=0x7fc59c114490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:44.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.148+0000 7fc5a1200700 1 --2- 172.21.15.27:0/619128135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59c1046d0 0x7fc59c114490 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc58400b3b0 tx=0x7fc584009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:44.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.148+0000 7fc5a1200700 1 -- 172.21.15.27:0/619128135 shutdown_connections 2024-04-16T05:47:44.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.148+0000 7fc5a1200700 1 --2- 172.21.15.27:0/619128135 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc58006ec00 0x7fc5800710b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.148+0000 7fc5a1200700 1 --2- 172.21.15.27:0/619128135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc59c103420 0x7fc59c113f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.148+0000 7fc5a1200700 1 --2- 172.21.15.27:0/619128135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc59c1046d0 0x7fc59c114490 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.148+0000 7fc5a1200700 1 -- 172.21.15.27:0/619128135 >> 172.21.15.27:0/619128135 conn(0x7fc59c072640 msgr2=0x7fc59c073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.148+0000 7fc5a1200700 1 -- 172.21.15.27:0/619128135 shutdown_connections 2024-04-16T05:47:44.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.148+0000 7fc5a1200700 1 -- 172.21.15.27:0/619128135 wait complete. 2024-04-16T05:47:44.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.303+0000 7f8e17400700 1 -- 172.21.15.27:0/2853206931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e10066f60 msgr2=0x7f8e10101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:44.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.303+0000 7f8e17400700 1 --2- 172.21.15.27:0/2853206931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e10066f60 0x7f8e10101de0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8e04004940 tx=0x7f8e04009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:44.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.303+0000 7f8e17400700 1 -- 172.21.15.27:0/2853206931 shutdown_connections 2024-04-16T05:47:44.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.303+0000 7f8e17400700 1 --2- 172.21.15.27:0/2853206931 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e10102320 0x7f8e10104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:44.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.303+0000 7f8e17400700 1 --2- 172.21.15.27:0/2853206931 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e10066f60 0x7f8e10101de0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:44.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.303+0000 7f8e17400700 1 -- 172.21.15.27:0/2853206931 >> 172.21.15.27:0/2853206931 conn(0x7f8e100f7450 msgr2=0x7f8e100f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:44.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.303+0000 7f8e17400700 1 -- 172.21.15.27:0/2853206931 shutdown_connections 2024-04-16T05:47:44.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.304+0000 7f8e17400700 1 -- 172.21.15.27:0/2853206931 wait complete. 2024-04-16T05:47:44.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.304+0000 7f8e17400700 1 Processor -- start 2024-04-16T05:47:44.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.304+0000 7f8e17400700 1 -- start start 2024-04-16T05:47:44.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.305+0000 7f8e17400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e10066f60 0x7f8e1010fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:44.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.305+0000 7f8e17400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e10102320 0x7f8e101103e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:44.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.305+0000 7f8e17400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e101109e0 con 0x7f8e10102320 2024-04-16T05:47:44.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.305+0000 7f8e17400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e10110b50 con 0x7f8e10066f60 2024-04-16T05:47:44.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.305+0000 7f8e0fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e10102320 0x7f8e101103e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:44.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.305+0000 7f8e0fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e10102320 0x7f8e101103e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50632/0 (socket says 172.21.15.27:50632) 2024-04-16T05:47:44.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.305+0000 7f8e0fe00700 1 -- 172.21.15.27:0/3980844140 learned_addr learned my addr 172.21.15.27:0/3980844140 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:47:44.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.306+0000 7f8e14a00700 1 --2- 172.21.15.27:0/3980844140 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e10066f60 0x7f8e1010fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:44.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.306+0000 7f8e0fe00700 1 -- 172.21.15.27:0/3980844140 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e10066f60 msgr2=0x7f8e1010fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:44.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.306+0000 7f8e0fe00700 1 --2- 172.21.15.27:0/3980844140 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e10066f60 0x7f8e1010fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:44.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.306+0000 7f8e0fe00700 1 -- 172.21.15.27:0/3980844140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e04005040 con 0x7f8e10102320 2024-04-16T05:47:44.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.306+0000 7f8e14a00700 1 --2- 172.21.15.27:0/3980844140 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e10066f60 0x7f8e1010fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T05:47:44.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.306+0000 7f8e0fe00700 1 --2- 172.21.15.27:0/3980844140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e10102320 0x7f8e101103e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc002a50 tx=0x7f8dfc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:44.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.306+0000 7f8e0cc00700 1 -- 172.21.15.27:0/3980844140 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8dfc007cc0 con 0x7f8e10102320 2024-04-16T05:47:44.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.306+0000 7f8e0cc00700 1 -- 172.21.15.27:0/3980844140 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8dfc00f040 con 0x7f8e10102320 2024-04-16T05:47:44.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.307+0000 7f8e17400700 1 -- 172.21.15.27:0/3980844140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e101155b0 con 0x7f8e10102320 2024-04-16T05:47:44.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.307+0000 7f8e0cc00700 1 -- 172.21.15.27:0/3980844140 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8dfc009640 con 0x7f8e10102320 2024-04-16T05:47:44.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.307+0000 7f8e17400700 1 -- 172.21.15.27:0/3980844140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e10115b70 con 0x7f8e10102320 2024-04-16T05:47:44.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.308+0000 7f8e17400700 1 -- 172.21.15.27:0/3980844140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e1004c710 con 0x7f8e10102320 2024-04-16T05:47:44.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.309+0000 7f8e0cc00700 1 -- 172.21.15.27:0/3980844140 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8dfc0097a0 con 0x7f8e10102320 2024-04-16T05:47:44.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.311+0000 7f8e0cc00700 1 --2- 172.21.15.27:0/3980844140 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8df806ec00 0x7f8df80710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:44.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.312+0000 7f8e0cc00700 1 -- 172.21.15.27:0/3980844140 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f8dfc08e7f0 con 0x7f8e10102320 2024-04-16T05:47:44.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.312+0000 7f8e14a00700 1 --2- 172.21.15.27:0/3980844140 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8df806ec00 0x7f8df80710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:44.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.313+0000 7f8e14a00700 1 --2- 172.21.15.27:0/3980844140 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8df806ec00 0x7f8df80710b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8e0400b3b0 tx=0x7f8e04009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:44.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.316+0000 7f8e0cc00700 1 -- 172.21.15.27:0/3980844140 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8dfc05af60 con 0x7f8e10102320 2024-04-16T05:47:44.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.605+0000 7f8e17400700 1 -- 172.21.15.27:0/3980844140 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f8e10115d90 con 0x7f8e10102320 2024-04-16T05:47:44.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.606+0000 7f8e0cc00700 1 -- 172.21.15.27:0/3980844140 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1880 (secure 0 0 0) 0x7f8dfc05aaf0 con 0x7f8e10102320 2024-04-16T05:47:44.606 INFO:teuthology.orchestra.run.smithi027.stdout:e16 2024-04-16T05:47:44.606 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:47:44.606 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:47:44.606 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:47:44.607 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:47:44.607 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:47:44.607 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:47:44.607 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 15 2024-04-16T05:47:44.607 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:47:44.607 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:47:44.607 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:40:32.701649+0000 2024-04-16T05:47:44.607 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:47:44.607 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:47:44.607 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:47:44.607 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:47:44.607 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:47:44.607 INFO:teuthology.orchestra.run.smithi027.stdout:max_xattr_size 65536 2024-04-16T05:47:44.607 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:47:44.607 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:47:44.607 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 46 2024-04-16T05:47:44.607 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:47:44.608 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:47:44.608 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:47:44.608 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=14462} 2024-04-16T05:47:44.608 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:47:44.608 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:47:44.608 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:47:44.608 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:47:44.608 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:47:44.608 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:47:44.608 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:47:44.608 INFO:teuthology.orchestra.run.smithi027.stdout:bal_rank_mask -1 2024-04-16T05:47:44.608 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:47:44.608 INFO:teuthology.orchestra.run.smithi027.stdout:qdb_cluster leader: 0 members: 2024-04-16T05:47:44.608 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:47:44.609 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:47:44.609 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:47:44.609 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:47:44.609 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:47:44.609 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{-1:14474} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2065690722,v1:172.21.15.27:6827/2065690722] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:47:44.609 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:24263} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:47:44.609 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:24273} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:47:44.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.609+0000 7f8e17400700 1 -- 172.21.15.27:0/3980844140 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8df806ec00 msgr2=0x7f8df80710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:44.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.609+0000 7f8e17400700 1 --2- 172.21.15.27:0/3980844140 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8df806ec00 0x7f8df80710b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8e0400b3b0 tx=0x7f8e04009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:44.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.609+0000 7f8e17400700 1 -- 172.21.15.27:0/3980844140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e10102320 msgr2=0x7f8e101103e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:44.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.609+0000 7f8e17400700 1 --2- 172.21.15.27:0/3980844140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e10102320 0x7f8e101103e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc002a50 tx=0x7f8dfc0046b0 comp rx=0 tx=0).stop 2024-04-16T05:47:44.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.609+0000 7f8e17400700 1 -- 172.21.15.27:0/3980844140 shutdown_connections 2024-04-16T05:47:44.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.609+0000 7f8e17400700 1 --2- 172.21.15.27:0/3980844140 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8df806ec00 0x7f8df80710b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:44.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.609+0000 7f8e17400700 1 --2- 172.21.15.27:0/3980844140 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8e10066f60 0x7f8e1010fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:44.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.609+0000 7f8e17400700 1 --2- 172.21.15.27:0/3980844140 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e10102320 0x7f8e101103e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:44.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.609+0000 7f8e17400700 1 -- 172.21.15.27:0/3980844140 >> 172.21.15.27:0/3980844140 conn(0x7f8e100f7450 msgr2=0x7f8e100f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:44.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.610+0000 7f8e17400700 1 -- 172.21.15.27:0/3980844140 shutdown_connections 2024-04-16T05:47:44.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.610+0000 7f8e17400700 1 -- 172.21.15.27:0/3980844140 wait complete. 2024-04-16T05:47:44.611 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2024-04-16T05:47:44.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.762+0000 7fadbe200700 1 -- 172.21.15.27:0/1854716452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadb80fce40 msgr2=0x7fadb80fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:44.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.762+0000 7fadbe200700 1 --2- 172.21.15.27:0/1854716452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadb80fce40 0x7fadb80fd250 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fada8004940 tx=0x7fada8009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:44.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.762+0000 7fadbe200700 1 -- 172.21.15.27:0/1854716452 shutdown_connections 2024-04-16T05:47:44.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.762+0000 7fadbe200700 1 --2- 172.21.15.27:0/1854716452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadb80fe040 0x7fadb80fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:44.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.762+0000 7fadbe200700 1 --2- 172.21.15.27:0/1854716452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadb80fce40 0x7fadb80fd250 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:44.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.762+0000 7fadbe200700 1 -- 172.21.15.27:0/1854716452 >> 172.21.15.27:0/1854716452 conn(0x7fadb80f8530 msgr2=0x7fadb80fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:44.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.762+0000 7fadbe200700 1 -- 172.21.15.27:0/1854716452 shutdown_connections 2024-04-16T05:47:44.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.762+0000 7fadbe200700 1 -- 172.21.15.27:0/1854716452 wait complete. 2024-04-16T05:47:44.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.763+0000 7fadbe200700 1 Processor -- start 2024-04-16T05:47:44.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.763+0000 7fadbe200700 1 -- start start 2024-04-16T05:47:44.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.764+0000 7fadbe200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadb80fce40 0x7fadb8071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:44.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.764+0000 7fadbe200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadb80fe040 0x7fadb8070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:44.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.764+0000 7fadbe200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb80708c0 con 0x7fadb80fce40 2024-04-16T05:47:44.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.764+0000 7fadbe200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb8070a30 con 0x7fadb80fe040 2024-04-16T05:47:44.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.764+0000 7fadb7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadb80fce40 0x7fadb8071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:44.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.764+0000 7fadb7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadb80fce40 0x7fadb8071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50652/0 (socket says 172.21.15.27:50652) 2024-04-16T05:47:44.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.764+0000 7fadb7400700 1 -- 172.21.15.27:0/3303438794 learned_addr learned my addr 172.21.15.27:0/3303438794 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:47:44.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.764+0000 7fadb6a00700 1 --2- 172.21.15.27:0/3303438794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadb80fe040 0x7fadb8070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:44.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.765+0000 7fadb7400700 1 -- 172.21.15.27:0/3303438794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadb80fe040 msgr2=0x7fadb8070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:44.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.765+0000 7fadb7400700 1 --2- 172.21.15.27:0/3303438794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadb80fe040 0x7fadb8070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:44.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.765+0000 7fadb7400700 1 -- 172.21.15.27:0/3303438794 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada8005040 con 0x7fadb80fce40 2024-04-16T05:47:44.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.765+0000 7fadb7400700 1 --2- 172.21.15.27:0/3303438794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadb80fce40 0x7fadb8071e40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fada800b970 tx=0x7fada8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:44.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.766+0000 7fadaf400700 1 -- 172.21.15.27:0/3303438794 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fada800e040 con 0x7fadb80fce40 2024-04-16T05:47:44.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.766+0000 7fadaf400700 1 -- 172.21.15.27:0/3303438794 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fada8004290 con 0x7fadb80fce40 2024-04-16T05:47:44.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.766+0000 7fadaf400700 1 -- 172.21.15.27:0/3303438794 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fada801c630 con 0x7fadb80fce40 2024-04-16T05:47:44.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.766+0000 7fadbe200700 1 -- 172.21.15.27:0/3303438794 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb8070cb0 con 0x7fadb80fce40 2024-04-16T05:47:44.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.766+0000 7fadbe200700 1 -- 172.21.15.27:0/3303438794 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb8115aa0 con 0x7fadb80fce40 2024-04-16T05:47:44.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.768+0000 7fadbe200700 1 -- 172.21.15.27:0/3303438794 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb805f5f0 con 0x7fadb80fce40 2024-04-16T05:47:44.767 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.769+0000 7fadaf400700 1 -- 172.21.15.27:0/3303438794 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fada800bbf0 con 0x7fadb80fce40 2024-04-16T05:47:44.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.769+0000 7fadaf400700 1 --2- 172.21.15.27:0/3303438794 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fad9c06ec10 0x7fad9c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:44.768 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.770+0000 7fadaf400700 1 -- 172.21.15.27:0/3303438794 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6736+0+0 (secure 0 0 0) 0x7fada808fbd0 con 0x7fadb80fce40 2024-04-16T05:47:44.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.770+0000 7fadb6a00700 1 --2- 172.21.15.27:0/3303438794 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fad9c06ec10 0x7fad9c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:44.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.771+0000 7fadb6a00700 1 --2- 172.21.15.27:0/3303438794 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fad9c06ec10 0x7fad9c0710c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fada4007fd0 tx=0x7fada4009f60 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:44.774 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:44.775+0000 7fadaf400700 1 -- 172.21.15.27:0/3303438794 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fada80589d0 con 0x7fadb80fce40 2024-04-16T05:47:44.774 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:44 smithi027 ceph-mon[93635]: from='client.34244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:47:44.774 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:44 smithi027 ceph-mon[93635]: pgmap v79: 65 pgs: 65 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:44.774 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:44 smithi027 ceph-mon[93635]: from='client.34248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:47:44.774 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:44.774 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:47:44.774 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:44 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/619128135' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:47:45.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:44 smithi122 ceph-mon[80930]: from='client.34244 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:47:45.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:44 smithi122 ceph-mon[80930]: pgmap v79: 65 pgs: 65 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:45.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:44 smithi122 ceph-mon[80930]: from='client.34248 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:47:45.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:44 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:45.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:44 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:47:45.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:44 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/619128135' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:47:45.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.029+0000 7fadbe200700 1 -- 172.21.15.27:0/3303438794 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fadb81023a0 con 0x7fad9c06ec10 2024-04-16T05:47:45.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.031+0000 7fadaf400700 1 -- 172.21.15.27:0/3303438794 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7fadb81023a0 con 0x7fad9c06ec10 2024-04-16T05:47:45.030 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:47:45.030 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T05:47:45.030 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:47:45.031 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:47:45.031 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [ 2024-04-16T05:47:45.031 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr", 2024-04-16T05:47:45.031 INFO:teuthology.orchestra.run.smithi027.stdout: "mon", 2024-04-16T05:47:45.031 INFO:teuthology.orchestra.run.smithi027.stdout: "crash" 2024-04-16T05:47:45.031 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-16T05:47:45.031 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "7/21 daemons upgraded", 2024-04-16T05:47:45.031 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Currently upgrading osd daemons", 2024-04-16T05:47:45.031 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:47:45.031 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:47:45.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.034+0000 7fadbe200700 1 -- 172.21.15.27:0/3303438794 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fad9c06ec10 msgr2=0x7fad9c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:45.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.034+0000 7fadbe200700 1 --2- 172.21.15.27:0/3303438794 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fad9c06ec10 0x7fad9c0710c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fada4007fd0 tx=0x7fada4009f60 comp rx=0 tx=0).stop 2024-04-16T05:47:45.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.034+0000 7fadbe200700 1 -- 172.21.15.27:0/3303438794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadb80fce40 msgr2=0x7fadb8071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:45.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.034+0000 7fadbe200700 1 --2- 172.21.15.27:0/3303438794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadb80fce40 0x7fadb8071e40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fada800b970 tx=0x7fada8009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:45.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.034+0000 7fadbe200700 1 -- 172.21.15.27:0/3303438794 shutdown_connections 2024-04-16T05:47:45.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.034+0000 7fadbe200700 1 --2- 172.21.15.27:0/3303438794 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fad9c06ec10 0x7fad9c0710c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:45.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.034+0000 7fadbe200700 1 --2- 172.21.15.27:0/3303438794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadb80fce40 0x7fadb8071e40 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:45.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.034+0000 7fadbe200700 1 --2- 172.21.15.27:0/3303438794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fadb80fe040 0x7fadb8070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:45.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.034+0000 7fadbe200700 1 -- 172.21.15.27:0/3303438794 >> 172.21.15.27:0/3303438794 conn(0x7fadb80f8530 msgr2=0x7fadb80f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:45.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.035+0000 7fadbe200700 1 -- 172.21.15.27:0/3303438794 shutdown_connections 2024-04-16T05:47:45.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.035+0000 7fadbe200700 1 -- 172.21.15.27:0/3303438794 wait complete. 2024-04-16T05:47:45.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.183+0000 7ffa20a00700 1 -- 172.21.15.27:0/3256129605 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa1400bcb0 con 0x7ffa240662f0 2024-04-16T05:47:45.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.184+0000 7ffa23200700 1 -- 172.21.15.27:0/3256129605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa240662f0 msgr2=0x7ffa240ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:45.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.184+0000 7ffa23200700 1 --2- 172.21.15.27:0/3256129605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa240662f0 0x7ffa240ff800 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ffa14004940 tx=0x7ffa14009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:45.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.184+0000 7ffa23200700 1 -- 172.21.15.27:0/3256129605 shutdown_connections 2024-04-16T05:47:45.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.184+0000 7ffa23200700 1 --2- 172.21.15.27:0/3256129605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa240ffd40 0x7ffa24102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:45.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.184+0000 7ffa23200700 1 --2- 172.21.15.27:0/3256129605 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa240662f0 0x7ffa240ff800 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:45.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.184+0000 7ffa23200700 1 -- 172.21.15.27:0/3256129605 >> 172.21.15.27:0/3256129605 conn(0x7ffa240f7270 msgr2=0x7ffa240f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:45.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.184+0000 7ffa23200700 1 -- 172.21.15.27:0/3256129605 shutdown_connections 2024-04-16T05:47:45.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.184+0000 7ffa23200700 1 -- 172.21.15.27:0/3256129605 wait complete. 2024-04-16T05:47:45.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.185+0000 7ffa23200700 1 Processor -- start 2024-04-16T05:47:45.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.185+0000 7ffa23200700 1 -- start start 2024-04-16T05:47:45.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.186+0000 7ffa23200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa240662f0 0x7ffa2410faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:45.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.186+0000 7ffa23200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa240ffd40 0x7ffa2410ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:45.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.186+0000 7ffa23200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa241105e0 con 0x7ffa240662f0 2024-04-16T05:47:45.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.186+0000 7ffa23200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa24110e30 con 0x7ffa240ffd40 2024-04-16T05:47:45.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.186+0000 7ffa21e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa240662f0 0x7ffa2410faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:45.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.186+0000 7ffa21e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa240662f0 0x7ffa2410faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50666/0 (socket says 172.21.15.27:50666) 2024-04-16T05:47:45.185 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.186+0000 7ffa21e00700 1 -- 172.21.15.27:0/2587988642 learned_addr learned my addr 172.21.15.27:0/2587988642 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:47:45.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.186+0000 7ffa21400700 1 --2- 172.21.15.27:0/2587988642 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa240ffd40 0x7ffa2410ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:45.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.186+0000 7ffa21e00700 1 -- 172.21.15.27:0/2587988642 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa240ffd40 msgr2=0x7ffa2410ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:45.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.186+0000 7ffa21e00700 1 --2- 172.21.15.27:0/2587988642 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa240ffd40 0x7ffa2410ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:45.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.187+0000 7ffa21e00700 1 -- 172.21.15.27:0/2587988642 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa14005040 con 0x7ffa240662f0 2024-04-16T05:47:45.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.187+0000 7ffa21e00700 1 --2- 172.21.15.27:0/2587988642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa240662f0 0x7ffa2410faa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffa14000c00 tx=0x7ffa14009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:45.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.187+0000 7ffa1a000700 1 -- 172.21.15.27:0/2587988642 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa1400e040 con 0x7ffa240662f0 2024-04-16T05:47:45.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.187+0000 7ffa1a000700 1 -- 172.21.15.27:0/2587988642 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa1400bcc0 con 0x7ffa240662f0 2024-04-16T05:47:45.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.187+0000 7ffa1a000700 1 -- 172.21.15.27:0/2587988642 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa1401c630 con 0x7ffa240662f0 2024-04-16T05:47:45.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.187+0000 7ffa23200700 1 -- 172.21.15.27:0/2587988642 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa241110b0 con 0x7ffa240662f0 2024-04-16T05:47:45.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.187+0000 7ffa23200700 1 -- 172.21.15.27:0/2587988642 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa241115f0 con 0x7ffa240662f0 2024-04-16T05:47:45.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.189+0000 7ffa23200700 1 -- 172.21.15.27:0/2587988642 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2405f5f0 con 0x7ffa240662f0 2024-04-16T05:47:45.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.190+0000 7ffa1a000700 1 -- 172.21.15.27:0/2587988642 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ffa14013980 con 0x7ffa240662f0 2024-04-16T05:47:45.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.191+0000 7ffa1a000700 1 --2- 172.21.15.27:0/2587988642 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ffa0806ec00 0x7ffa080710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:47:45.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.191+0000 7ffa1a000700 1 -- 172.21.15.27:0/2587988642 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6736+0+0 (secure 0 0 0) 0x7ffa1408fd00 con 0x7ffa240662f0 2024-04-16T05:47:45.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.191+0000 7ffa21400700 1 --2- 172.21.15.27:0/2587988642 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ffa0806ec00 0x7ffa080710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:47:45.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.192+0000 7ffa21400700 1 --2- 172.21.15.27:0/2587988642 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ffa0806ec00 0x7ffa080710b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffa10004990 tx=0x7ffa10009ee0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:47:45.199 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.200+0000 7ffa1a000700 1 -- 172.21.15.27:0/2587988642 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ffa1405c570 con 0x7ffa240662f0 2024-04-16T05:47:45.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.511+0000 7ffa23200700 1 -- 172.21.15.27:0/2587988642 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ffa2404bed0 con 0x7ffa240662f0 2024-04-16T05:47:45.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.512+0000 7ffa1a000700 1 -- 172.21.15.27:0/2587988642 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7ffa1405c100 con 0x7ffa240662f0 2024-04-16T05:47:45.511 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:47:45.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.514+0000 7ffa23200700 1 -- 172.21.15.27:0/2587988642 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ffa0806ec00 msgr2=0x7ffa080710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:45.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.514+0000 7ffa23200700 1 --2- 172.21.15.27:0/2587988642 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ffa0806ec00 0x7ffa080710b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffa10004990 tx=0x7ffa10009ee0 comp rx=0 tx=0).stop 2024-04-16T05:47:45.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.514+0000 7ffa23200700 1 -- 172.21.15.27:0/2587988642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa240662f0 msgr2=0x7ffa2410faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:47:45.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.514+0000 7ffa23200700 1 --2- 172.21.15.27:0/2587988642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa240662f0 0x7ffa2410faa0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffa14000c00 tx=0x7ffa14009e30 comp rx=0 tx=0).stop 2024-04-16T05:47:45.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.515+0000 7ffa23200700 1 -- 172.21.15.27:0/2587988642 shutdown_connections 2024-04-16T05:47:45.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.515+0000 7ffa23200700 1 --2- 172.21.15.27:0/2587988642 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ffa0806ec00 0x7ffa080710b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:45.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.515+0000 7ffa23200700 1 --2- 172.21.15.27:0/2587988642 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffa240662f0 0x7ffa2410faa0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:45.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.515+0000 7ffa23200700 1 --2- 172.21.15.27:0/2587988642 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffa240ffd40 0x7ffa2410ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:47:45.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.515+0000 7ffa23200700 1 -- 172.21.15.27:0/2587988642 >> 172.21.15.27:0/2587988642 conn(0x7ffa240f7270 msgr2=0x7ffa240f7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:47:45.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.515+0000 7ffa23200700 1 -- 172.21.15.27:0/2587988642 shutdown_connections 2024-04-16T05:47:45.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:47:45.515+0000 7ffa23200700 1 -- 172.21.15.27:0/2587988642 wait complete. 2024-04-16T05:47:45.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:45 smithi027 ceph-mon[93635]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:47:45.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:45 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/3980844140' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:47:45.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:45 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/2587988642' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:47:45.958 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 182, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104442 2024-04-16T05:47:45.958 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 182, success, outsize: 120 2024-04-16T05:47:45.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 184, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104535 2024-04-16T05:47:45.982 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 184, success, outsize: 120 2024-04-16T05:47:46.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 186, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104631 2024-04-16T05:47:46.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 186, success, outsize: 120 2024-04-16T05:47:46.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:45 smithi122 ceph-mon[80930]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:47:46.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:45 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/3980844140' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:47:46.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:45 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/2587988642' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:47:46.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 188, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104733 2024-04-16T05:47:46.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 188, success, outsize: 120 2024-04-16T05:47:46.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 190, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104832 2024-04-16T05:47:46.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 190, success, outsize: 120 2024-04-16T05:47:46.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 192, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104925 2024-04-16T05:47:46.082 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 192, success, outsize: 120 2024-04-16T05:47:46.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 194, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105021 2024-04-16T05:47:46.107 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 194, success, outsize: 120 2024-04-16T05:47:46.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 196, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105123 2024-04-16T05:47:46.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 196, success, outsize: 120 2024-04-16T05:47:46.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 198, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105222 2024-04-16T05:47:46.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 198, success, outsize: 120 2024-04-16T05:47:46.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 200, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105315 2024-04-16T05:47:46.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 200, success, outsize: 120 2024-04-16T05:47:46.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 202, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105411 2024-04-16T05:47:46.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 202, success, outsize: 120 2024-04-16T05:47:46.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 204, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105513 2024-04-16T05:47:46.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 204, success, outsize: 120 2024-04-16T05:47:46.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 206, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105612 2024-04-16T05:47:46.266 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 206, success, outsize: 120 2024-04-16T05:47:46.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 208, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105705 2024-04-16T05:47:46.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 208, success, outsize: 120 2024-04-16T05:47:46.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 210, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105801 2024-04-16T05:47:46.321 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 210, success, outsize: 120 2024-04-16T05:47:46.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 212, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105903 2024-04-16T05:47:46.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 212, success, outsize: 120 2024-04-16T05:47:46.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 214, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106002 2024-04-16T05:47:46.380 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 214, success, outsize: 120 2024-04-16T05:47:46.412 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 216, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106095 2024-04-16T05:47:46.412 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 216, success, outsize: 120 2024-04-16T05:47:46.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 218, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106191 2024-04-16T05:47:46.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 218, success, outsize: 120 2024-04-16T05:47:46.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 220, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106293 2024-04-16T05:47:46.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 220, success, outsize: 120 2024-04-16T05:47:46.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 222, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106392 2024-04-16T05:47:46.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 222, success, outsize: 120 2024-04-16T05:47:46.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 224, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106485 2024-04-16T05:47:46.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 224, success, outsize: 120 2024-04-16T05:47:46.541 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 226, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106581 2024-04-16T05:47:46.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 226, success, outsize: 120 2024-04-16T05:47:46.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 228, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106683 2024-04-16T05:47:46.568 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 228, success, outsize: 120 2024-04-16T05:47:46.590 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 230, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106782 2024-04-16T05:47:46.591 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 230, success, outsize: 120 2024-04-16T05:47:46.611 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 232, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106875 2024-04-16T05:47:46.611 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 232, success, outsize: 120 2024-04-16T05:47:46.632 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 234, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106971 2024-04-16T05:47:46.632 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 234, success, outsize: 120 2024-04-16T05:47:46.656 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 236, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107073 2024-04-16T05:47:46.656 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 236, success, outsize: 120 2024-04-16T05:47:46.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:46 smithi027 ceph-mon[93635]: pgmap v80: 65 pgs: 65 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:46.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:46 smithi027 ceph-mon[93635]: from='client.34264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:47:47.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:46 smithi122 ceph-mon[80930]: pgmap v80: 65 pgs: 65 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:47.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:46 smithi122 ceph-mon[80930]: from='client.34264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:47:48.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:48 smithi027 ceph-mon[93635]: pgmap v81: 65 pgs: 65 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:48.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:48 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:48.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:48 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:48.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:48 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:47:48.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:48 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:47:48.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:48 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:48.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:48 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:47:48.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:48 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:47:48.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:48 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:47:48.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:48 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:47:48.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:48 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-16T05:47:49.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:48 smithi122 ceph-mon[80930]: pgmap v81: 65 pgs: 65 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:49.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:48 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:49.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:48 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:49.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:48 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:47:49.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:48 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:47:49.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:48 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:49.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:48 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:47:49.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:48 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:47:49.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:48 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:47:49.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:48 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:47:49.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:48 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-16T05:47:49.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:49 smithi027 ceph-mon[93635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-16T05:47:49.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:49 smithi027 ceph-mon[93635]: Upgrade: osd.1 is safe to restart 2024-04-16T05:47:49.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:49 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:50.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:49 smithi122 ceph-mon[80930]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-16T05:47:50.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:49 smithi122 ceph-mon[80930]: Upgrade: osd.1 is safe to restart 2024-04-16T05:47:50.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:49 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:51.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:50 smithi122 ceph-mon[80930]: pgmap v82: 65 pgs: 65 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:51.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:50 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:51.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:50 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T05:47:51.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:50 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:47:51.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:50 smithi027 ceph-mon[93635]: pgmap v82: 65 pgs: 65 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:51.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:50 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:47:51.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:50 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-16T05:47:51.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:50 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:47:51.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:51 smithi027 ceph-mon[93635]: Upgrade: Updating osd.1 2024-04-16T05:47:51.601 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:51 smithi027 ceph-mon[93635]: Deploying daemon osd.1 on smithi027 2024-04-16T05:47:52.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:51 smithi122 ceph-mon[80930]: Upgrade: Updating osd.1 2024-04-16T05:47:52.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:51 smithi122 ceph-mon[80930]: Deploying daemon osd.1 on smithi027 2024-04-16T05:47:52.923 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:52 smithi122 ceph-mon[80930]: pgmap v83: 65 pgs: 65 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:53.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:52 smithi027 ceph-mon[93635]: pgmap v83: 65 pgs: 65 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:55.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:54 smithi122 ceph-mon[80930]: pgmap v84: 65 pgs: 65 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:55.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:54 smithi027 ceph-mon[93635]: pgmap v84: 65 pgs: 65 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:56.057 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:47:55 smithi027 systemd[1]: Stopping Ceph osd.1 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T05:47:56.557 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:47:56 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1[55206]: 2024-04-16T05:47:56.059+0000 7f31be600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T05:47:56.557 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:47:56 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1[55206]: 2024-04-16T05:47:56.059+0000 7f31be600700 -1 osd.1 58 *** Got signal Terminated *** 2024-04-16T05:47:56.557 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:47:56 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1[55206]: 2024-04-16T05:47:56.059+0000 7f31be600700 -1 osd.1 58 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T05:47:57.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:56 smithi122 ceph-mon[80930]: pgmap v85: 65 pgs: 65 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:57.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:56 smithi122 ceph-mon[80930]: osd.1 marked itself down and dead 2024-04-16T05:47:57.045 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:56 smithi027 ceph-mon[93635]: pgmap v85: 65 pgs: 65 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:57.045 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:56 smithi027 ceph-mon[93635]: osd.1 marked itself down and dead 2024-04-16T05:47:57.306 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:47:57 smithi027 podman[107521]: 2024-04-16 05:47:57.047038472 +0000 UTC m=+1.141203338 container died e146295f65004edc827f2417f4d9acab27d26874e7019139576bf26fcbf80c52 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-64502-quincy-3-45317f9, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409) 2024-04-16T05:47:57.760 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:57 smithi027 ceph-mon[93635]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T05:47:57.760 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:57 smithi027 ceph-mon[93635]: osdmap e59: 6 total, 5 up, 6 in 2024-04-16T05:47:57.760 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:47:57 smithi027 podman[107521]: 2024-04-16 05:47:57.493528025 +0000 UTC m=+1.587692932 container cleanup e146295f65004edc827f2417f4d9acab27d26874e7019139576bf26fcbf80c52 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True) 2024-04-16T05:47:57.760 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:47:57 smithi027 bash[107521]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1 2024-04-16T05:47:58.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:57 smithi122 ceph-mon[80930]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T05:47:58.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:57 smithi122 ceph-mon[80930]: osdmap e59: 6 total, 5 up, 6 in 2024-04-16T05:47:58.056 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:47:57 smithi027 podman[107539]: 2024-04-16 05:47:57.761628312 +0000 UTC m=+0.694382566 container remove e146295f65004edc827f2417f4d9acab27d26874e7019139576bf26fcbf80c52 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-64502-quincy-3-45317f9, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-16T05:47:58.719 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:58 smithi027 ceph-mon[93635]: pgmap v87: 65 pgs: 11 stale+active+clean, 54 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:58.719 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:47:58 smithi027 ceph-mon[93635]: osdmap e60: 6 total, 5 up, 6 in 2024-04-16T05:47:58.719 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:47:58 smithi027 podman[107630]: 2024-04-16 05:47:58.57063835 +0000 UTC m=+0.043679333 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T05:47:58.719 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:47:58 smithi027 podman[107630]: 2024-04-16 05:47:58.720971697 +0000 UTC m=+0.194012715 container create 21d41cd63815b0ec5d1741f38e078b66f5d62b56602f002d0cef23cb79b7455d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-16T05:47:59.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:58 smithi122 ceph-mon[80930]: pgmap v87: 65 pgs: 11 stale+active+clean, 54 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:47:59.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:47:58 smithi122 ceph-mon[80930]: osdmap e60: 6 total, 5 up, 6 in 2024-04-16T05:47:59.557 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:47:59 smithi027 podman[107630]: 2024-04-16 05:47:59.124803074 +0000 UTC m=+0.597844099 container init 21d41cd63815b0ec5d1741f38e078b66f5d62b56602f002d0cef23cb79b7455d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-16T05:47:59.557 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:47:59 smithi027 podman[107630]: 2024-04-16 05:47:59.137224655 +0000 UTC m=+0.610265706 container start 21d41cd63815b0ec5d1741f38e078b66f5d62b56602f002d0cef23cb79b7455d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240409) 2024-04-16T05:47:59.557 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:47:59 smithi027 podman[107630]: 2024-04-16 05:47:59.290379655 +0000 UTC m=+0.763420716 container attach 21d41cd63815b0ec5d1741f38e078b66f5d62b56602f002d0cef23cb79b7455d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-04-16T05:48:00.001 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:47:59 smithi027 podman[107630]: 2024-04-16 05:47:59.617334724 +0000 UTC m=+1.090375749 container died 21d41cd63815b0ec5d1741f38e078b66f5d62b56602f002d0cef23cb79b7455d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_CLEAN=True) 2024-04-16T05:48:00.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:00 smithi122 ceph-mon[80930]: pgmap v89: 65 pgs: 11 stale+active+clean, 54 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:00.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:00.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:48:00.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:00 smithi027 ceph-mon[93635]: pgmap v89: 65 pgs: 11 stale+active+clean, 54 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:00.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:00.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:48:00.807 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:00 smithi027 podman[107656]: 2024-04-16 05:48:00.468988957 +0000 UTC m=+0.832396543 container remove 21d41cd63815b0ec5d1741f38e078b66f5d62b56602f002d0cef23cb79b7455d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T05:48:00.807 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:00 smithi027 systemd[1]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.1.service: Deactivated successfully. 2024-04-16T05:48:00.807 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:00 smithi027 systemd[1]: Stopped Ceph osd.1 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968. 2024-04-16T05:48:00.807 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:00 smithi027 systemd[1]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.1.service: Consumed 14.910s CPU time. 2024-04-16T05:48:01.278 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:01 smithi027 ceph-mon[93635]: Health check failed: Degraded data redundancy: 10/72 objects degraded (13.889%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T05:48:01.279 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:00 smithi027 systemd[1]: Starting Ceph osd.1 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T05:48:01.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:01 smithi122 ceph-mon[80930]: Health check failed: Degraded data redundancy: 10/72 objects degraded (13.889%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T05:48:01.557 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:01 smithi027 podman[107758]: 2024-04-16 05:48:01.187038051 +0000 UTC m=+0.035113609 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T05:48:01.557 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:01 smithi027 podman[107758]: 2024-04-16 05:48:01.328755 +0000 UTC m=+0.176830619 container create 180b3ebeea909ebd1ed9c53c9149b82d3aa97c5c1bd897d731d7751a91f31a6d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-activate, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-16T05:48:02.057 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:01 smithi027 podman[107758]: 2024-04-16 05:48:01.625495924 +0000 UTC m=+0.473571533 container init 180b3ebeea909ebd1ed9c53c9149b82d3aa97c5c1bd897d731d7751a91f31a6d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=) 2024-04-16T05:48:02.057 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:01 smithi027 podman[107758]: 2024-04-16 05:48:01.639087751 +0000 UTC m=+0.487163325 container start 180b3ebeea909ebd1ed9c53c9149b82d3aa97c5c1bd897d731d7751a91f31a6d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T05:48:02.057 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:01 smithi027 podman[107758]: 2024-04-16 05:48:01.756482312 +0000 UTC m=+0.604557880 container attach 180b3ebeea909ebd1ed9c53c9149b82d3aa97c5c1bd897d731d7751a91f31a6d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-activate, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS) 2024-04-16T05:48:02.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 238, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107852 2024-04-16T05:48:02.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 238, success, outsize: 120 2024-04-16T05:48:02.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:02 smithi122 ceph-mon[80930]: pgmap v90: 65 pgs: 5 active+undersized+degraded, 2 stale+active+clean, 24 active+undersized, 34 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T05:48:02.554 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:02 smithi027 ceph-mon[93635]: pgmap v90: 65 pgs: 5 active+undersized+degraded, 2 stale+active+clean, 24 active+undersized, 34 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T05:48:02.808 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:02 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-activate[107773]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T05:48:02.808 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:02 smithi027 bash[107758]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T05:48:02.808 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:02 smithi027 bash[107758]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-16T05:48:02.808 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:02 smithi027 bash[107758]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-16T05:48:02.808 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:02 smithi027 bash[107758]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-16T05:48:02.808 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:02 smithi027 bash[107758]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-16T05:48:02.808 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:02 smithi027 bash[107758]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T05:48:02.808 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:02 smithi027 bash[107758]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-16T05:48:02.808 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:02 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-activate[107773]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-16T05:48:02.808 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:02 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-activate[107773]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-16T05:48:02.808 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:02 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-activate[107773]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-16T05:48:02.808 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:02 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-activate[107773]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-16T05:48:02.809 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:02 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-activate[107773]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T05:48:02.809 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:02 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-activate[107773]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-16T05:48:02.809 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:02 smithi027 podman[107758]: 2024-04-16 05:48:02.62263128 +0000 UTC m=+1.470706876 container died 180b3ebeea909ebd1ed9c53c9149b82d3aa97c5c1bd897d731d7751a91f31a6d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-activate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-16T05:48:03.806 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:03 smithi027 podman[107896]: 2024-04-16 05:48:03.410404484 +0000 UTC m=+0.765273587 container remove 180b3ebeea909ebd1ed9c53c9149b82d3aa97c5c1bd897d731d7751a91f31a6d (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-activate, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-04-16T05:48:04.066 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:03 smithi027 podman[107957]: 2024-04-16 05:48:03.718362792 +0000 UTC m=+0.036724018 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T05:48:04.066 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:03 smithi027 podman[107957]: 2024-04-16 05:48:03.836169357 +0000 UTC m=+0.154530619 container create 3a6ab1fc6e79d4c1f1c36a51cc12594bf9ac7084b927a2a536cee52c65dfa954 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T05:48:04.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:04 smithi122 ceph-mon[80930]: pgmap v91: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T05:48:04.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:04 smithi027 ceph-mon[93635]: pgmap v91: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T05:48:04.556 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:04 smithi027 podman[107957]: 2024-04-16 05:48:04.148295849 +0000 UTC m=+0.466657117 container init 3a6ab1fc6e79d4c1f1c36a51cc12594bf9ac7084b927a2a536cee52c65dfa954 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.build-date=20240409) 2024-04-16T05:48:04.557 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:04 smithi027 podman[107957]: 2024-04-16 05:48:04.160909381 +0000 UTC m=+0.479270670 container start 3a6ab1fc6e79d4c1f1c36a51cc12594bf9ac7084b927a2a536cee52c65dfa954 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-16T05:48:04.557 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:04 smithi027 ceph-osd[107975]: -- 172.21.15.27:0/1298457009 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5610d67e3680 con 0x5610d67ad000 2024-04-16T05:48:04.557 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:04 smithi027 bash[107957]: 3a6ab1fc6e79d4c1f1c36a51cc12594bf9ac7084b927a2a536cee52c65dfa954 2024-04-16T05:48:04.557 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:04 smithi027 systemd[1]: Started Ceph osd.1 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968. 2024-04-16T05:48:05.056 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:04 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1[107971]: 2024-04-16T05:48:04.740+0000 7f8b226497c0 -1 Falling back to public interface 2024-04-16T05:48:05.766 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:05 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:05.766 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:05 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:05.766 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:05 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:48:05.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:05 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:05.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:05 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:05.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:05 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:48:06.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:06 smithi122 ceph-mon[80930]: pgmap v92: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T05:48:06.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:06 smithi027 ceph-mon[93635]: pgmap v92: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T05:48:08.659 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:08 smithi027 ceph-mon[93635]: pgmap v93: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T05:48:08.659 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:08 smithi027 ceph-mon[93635]: from='osd.1 [v2:172.21.15.27:6810/2379867635,v1:172.21.15.27:6811/2379867635]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-16T05:48:08.659 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:08 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1[107971]: 2024-04-16T05:48:08.386+0000 7f8b226497c0 -1 osd.1 58 log_to_monitors true 2024-04-16T05:48:08.659 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 05:48:08 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1[107971]: 2024-04-16T05:48:08.457+0000 7f8b11a00700 -1 osd.1 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T05:48:08.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:08 smithi122 ceph-mon[80930]: pgmap v93: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 1.4 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T05:48:08.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:08 smithi122 ceph-mon[80930]: from='osd.1 [v2:172.21.15.27:6810/2379867635,v1:172.21.15.27:6811/2379867635]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-16T05:48:09.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:09 smithi122 ceph-mon[80930]: from='osd.1 [v2:172.21.15.27:6810/2379867635,v1:172.21.15.27:6811/2379867635]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-16T05:48:09.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:09 smithi122 ceph-mon[80930]: osdmap e61: 6 total, 5 up, 6 in 2024-04-16T05:48:09.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:09 smithi122 ceph-mon[80930]: from='osd.1 [v2:172.21.15.27:6810/2379867635,v1:172.21.15.27:6811/2379867635]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-16T05:48:09.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:09 smithi027 ceph-mon[93635]: from='osd.1 [v2:172.21.15.27:6810/2379867635,v1:172.21.15.27:6811/2379867635]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-16T05:48:09.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:09 smithi027 ceph-mon[93635]: osdmap e61: 6 total, 5 up, 6 in 2024-04-16T05:48:09.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:09 smithi027 ceph-mon[93635]: from='osd.1 [v2:172.21.15.27:6810/2379867635,v1:172.21.15.27:6811/2379867635]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-16T05:48:10.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:10 smithi122 ceph-mon[80930]: pgmap v95: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T05:48:10.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:10 smithi122 ceph-mon[80930]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T05:48:10.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:10 smithi122 ceph-mon[80930]: osd.1 [v2:172.21.15.27:6810/2379867635,v1:172.21.15.27:6811/2379867635] boot 2024-04-16T05:48:10.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:10 smithi122 ceph-mon[80930]: osdmap e62: 6 total, 6 up, 6 in 2024-04-16T05:48:10.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:10 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T05:48:10.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:10 smithi122 ceph-mon[80930]: osdmap e63: 6 total, 6 up, 6 in 2024-04-16T05:48:10.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:10 smithi027 ceph-mon[93635]: pgmap v95: 65 pgs: 5 active+undersized+degraded, 27 active+undersized, 33 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T05:48:10.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:10 smithi027 ceph-mon[93635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T05:48:10.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:10 smithi027 ceph-mon[93635]: osd.1 [v2:172.21.15.27:6810/2379867635,v1:172.21.15.27:6811/2379867635] boot 2024-04-16T05:48:10.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:10 smithi027 ceph-mon[93635]: osdmap e62: 6 total, 6 up, 6 in 2024-04-16T05:48:10.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:10 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-16T05:48:10.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:10 smithi027 ceph-mon[93635]: osdmap e63: 6 total, 6 up, 6 in 2024-04-16T05:48:11.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:11 smithi122 ceph-mon[80930]: Health check update: Degraded data redundancy: 4/72 objects degraded (5.556%), 3 pgs degraded (PG_DEGRADED) 2024-04-16T05:48:11.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:11 smithi027 ceph-mon[93635]: Health check update: Degraded data redundancy: 4/72 objects degraded (5.556%), 3 pgs degraded (PG_DEGRADED) 2024-04-16T05:48:12.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:12 smithi122 ceph-mon[80930]: pgmap v98: 65 pgs: 20 peering, 3 active+undersized+degraded, 9 active+undersized, 33 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 4/72 objects degraded (5.556%) 2024-04-16T05:48:12.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:12 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:12.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:12 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:12.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:12 smithi027 ceph-mon[93635]: pgmap v98: 65 pgs: 20 peering, 3 active+undersized+degraded, 9 active+undersized, 33 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 4/72 objects degraded (5.556%) 2024-04-16T05:48:12.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:12 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:12.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:12 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:14.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:14 smithi027 ceph-mon[93635]: pgmap v99: 65 pgs: 20 peering, 3 active+undersized+degraded, 8 active+undersized, 34 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 4/72 objects degraded (5.556%) 2024-04-16T05:48:14.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:14.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:14.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:14.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:48:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:14 smithi122 ceph-mon[80930]: pgmap v99: 65 pgs: 20 peering, 3 active+undersized+degraded, 8 active+undersized, 34 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 4/72 objects degraded (5.556%) 2024-04-16T05:48:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:15.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:15.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:48:15.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.687+0000 7f2c1f400700 1 -- 172.21.15.27:0/1626415687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c18103420 msgr2=0x7f2c18103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:15.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.687+0000 7f2c1f400700 1 --2- 172.21.15.27:0/1626415687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c18103420 0x7f2c18103830 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c004940 tx=0x7f2c0c009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:15.687 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:15 smithi027 ceph-mon[93635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/72 objects degraded (5.556%), 3 pgs degraded) 2024-04-16T05:48:15.687 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:15 smithi027 ceph-mon[93635]: Cluster is now healthy 2024-04-16T05:48:15.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.687+0000 7f2c1f400700 1 -- 172.21.15.27:0/1626415687 shutdown_connections 2024-04-16T05:48:15.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.687+0000 7f2c1f400700 1 --2- 172.21.15.27:0/1626415687 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c181046d0 0x7f2c18104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:15.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.687+0000 7f2c1f400700 1 --2- 172.21.15.27:0/1626415687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c18103420 0x7f2c18103830 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:15.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.687+0000 7f2c1f400700 1 -- 172.21.15.27:0/1626415687 >> 172.21.15.27:0/1626415687 conn(0x7f2c18072640 msgr2=0x7f2c18074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:15.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.687+0000 7f2c1f400700 1 -- 172.21.15.27:0/1626415687 shutdown_connections 2024-04-16T05:48:15.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.687+0000 7f2c1f400700 1 -- 172.21.15.27:0/1626415687 wait complete. 2024-04-16T05:48:15.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.688+0000 7f2c1f400700 1 Processor -- start 2024-04-16T05:48:15.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.688+0000 7f2c1f400700 1 -- start start 2024-04-16T05:48:15.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.689+0000 7f2c1f400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c18103420 0x7f2c18113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:15.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.689+0000 7f2c1f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c181046d0 0x7f2c181144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:15.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.689+0000 7f2c1f400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c18114aa0 con 0x7f2c181046d0 2024-04-16T05:48:15.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.689+0000 7f2c1f400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c18114c10 con 0x7f2c18103420 2024-04-16T05:48:15.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.689+0000 7f2c17e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c181046d0 0x7f2c181144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:15.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.689+0000 7f2c17e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c181046d0 0x7f2c181144a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40372/0 (socket says 172.21.15.27:40372) 2024-04-16T05:48:15.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.689+0000 7f2c17e00700 1 -- 172.21.15.27:0/2966256528 learned_addr learned my addr 172.21.15.27:0/2966256528 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:48:15.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.690+0000 7f2c1ca00700 1 --2- 172.21.15.27:0/2966256528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c18103420 0x7f2c18113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:15.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.690+0000 7f2c17e00700 1 -- 172.21.15.27:0/2966256528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c18103420 msgr2=0x7f2c18113f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:15.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.690+0000 7f2c17e00700 1 --2- 172.21.15.27:0/2966256528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c18103420 0x7f2c18113f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:15.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.690+0000 7f2c17e00700 1 -- 172.21.15.27:0/2966256528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0c005040 con 0x7f2c181046d0 2024-04-16T05:48:15.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.690+0000 7f2c17e00700 1 --2- 172.21.15.27:0/2966256528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c181046d0 0x7f2c181144a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2c04002a50 tx=0x7f2c040046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:15.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.690+0000 7f2c14c00700 1 -- 172.21.15.27:0/2966256528 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c04007cc0 con 0x7f2c181046d0 2024-04-16T05:48:15.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.690+0000 7f2c14c00700 1 -- 172.21.15.27:0/2966256528 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c0400f040 con 0x7f2c181046d0 2024-04-16T05:48:15.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.690+0000 7f2c1f400700 1 -- 172.21.15.27:0/2966256528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c18119670 con 0x7f2c181046d0 2024-04-16T05:48:15.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.690+0000 7f2c14c00700 1 -- 172.21.15.27:0/2966256528 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c04009640 con 0x7f2c181046d0 2024-04-16T05:48:15.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.691+0000 7f2c1f400700 1 -- 172.21.15.27:0/2966256528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c18119c30 con 0x7f2c181046d0 2024-04-16T05:48:15.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.692+0000 7f2c1f400700 1 -- 172.21.15.27:0/2966256528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c1804c710 con 0x7f2c181046d0 2024-04-16T05:48:15.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.693+0000 7f2c14c00700 1 -- 172.21.15.27:0/2966256528 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2c040097a0 con 0x7f2c181046d0 2024-04-16T05:48:15.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.694+0000 7f2c14c00700 1 --2- 172.21.15.27:0/2966256528 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2c0006ec00 0x7f2c000710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:15.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.694+0000 7f2c14c00700 1 -- 172.21.15.27:0/2966256528 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f2c0408e7f0 con 0x7f2c181046d0 2024-04-16T05:48:15.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.695+0000 7f2c1ca00700 1 --2- 172.21.15.27:0/2966256528 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2c0006ec00 0x7f2c000710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:15.694 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.695+0000 7f2c1ca00700 1 --2- 172.21.15.27:0/2966256528 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2c0006ec00 0x7f2c000710b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c00b3b0 tx=0x7f2c0c009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:15.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.699+0000 7f2c14c00700 1 -- 172.21.15.27:0/2966256528 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2c0405af60 con 0x7f2c181046d0 2024-04-16T05:48:15.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.954+0000 7f2c1f400700 1 -- 172.21.15.27:0/2966256528 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c18108a60 con 0x7f2c0006ec00 2024-04-16T05:48:15.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.957+0000 7f2c14c00700 1 -- 172.21.15.27:0/2966256528 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f2c18108a60 con 0x7f2c0006ec00 2024-04-16T05:48:15.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.960+0000 7f2c1f400700 1 -- 172.21.15.27:0/2966256528 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2c0006ec00 msgr2=0x7f2c000710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:15.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.960+0000 7f2c1f400700 1 --2- 172.21.15.27:0/2966256528 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2c0006ec00 0x7f2c000710b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c00b3b0 tx=0x7f2c0c009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:15.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.960+0000 7f2c1f400700 1 -- 172.21.15.27:0/2966256528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c181046d0 msgr2=0x7f2c181144a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:15.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.960+0000 7f2c1f400700 1 --2- 172.21.15.27:0/2966256528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c181046d0 0x7f2c181144a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2c04002a50 tx=0x7f2c040046b0 comp rx=0 tx=0).stop 2024-04-16T05:48:15.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.960+0000 7f2c1f400700 1 -- 172.21.15.27:0/2966256528 shutdown_connections 2024-04-16T05:48:15.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.960+0000 7f2c1f400700 1 --2- 172.21.15.27:0/2966256528 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2c0006ec00 0x7f2c000710b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:15.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.960+0000 7f2c1f400700 1 --2- 172.21.15.27:0/2966256528 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c18103420 0x7f2c18113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:15.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.960+0000 7f2c1f400700 1 --2- 172.21.15.27:0/2966256528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c181046d0 0x7f2c181144a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:15.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.960+0000 7f2c1f400700 1 -- 172.21.15.27:0/2966256528 >> 172.21.15.27:0/2966256528 conn(0x7f2c18072640 msgr2=0x7f2c18073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:15.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.960+0000 7f2c1f400700 1 -- 172.21.15.27:0/2966256528 shutdown_connections 2024-04-16T05:48:15.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:15.960+0000 7f2c1f400700 1 -- 172.21.15.27:0/2966256528 wait complete. 2024-04-16T05:48:15.976 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:48:16.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:15 smithi122 ceph-mon[80930]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/72 objects degraded (5.556%), 3 pgs degraded) 2024-04-16T05:48:16.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:15 smithi122 ceph-mon[80930]: Cluster is now healthy 2024-04-16T05:48:16.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.115+0000 7f7a5bc00700 1 -- 172.21.15.27:0/1064905350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a540ff020 msgr2=0x7f7a540ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:16.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.115+0000 7f7a5bc00700 1 --2- 172.21.15.27:0/1064905350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a540ff020 0x7f7a540ff430 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7a48004940 tx=0x7f7a48009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:16.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.115+0000 7f7a5bc00700 1 -- 172.21.15.27:0/1064905350 shutdown_connections 2024-04-16T05:48:16.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.115+0000 7f7a5bc00700 1 --2- 172.21.15.27:0/1064905350 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a541002d0 0x7f7a54100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:16.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.115+0000 7f7a5bc00700 1 --2- 172.21.15.27:0/1064905350 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a540ff020 0x7f7a540ff430 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:16.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.115+0000 7f7a5bc00700 1 -- 172.21.15.27:0/1064905350 >> 172.21.15.27:0/1064905350 conn(0x7f7a540fa950 msgr2=0x7f7a540fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:16.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.115+0000 7f7a5bc00700 1 -- 172.21.15.27:0/1064905350 shutdown_connections 2024-04-16T05:48:16.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.115+0000 7f7a5bc00700 1 -- 172.21.15.27:0/1064905350 wait complete. 2024-04-16T05:48:16.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.116+0000 7f7a5bc00700 1 Processor -- start 2024-04-16T05:48:16.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.117+0000 7f7a5bc00700 1 -- start start 2024-04-16T05:48:16.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.117+0000 7f7a5bc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a540ff020 0x7f7a5410fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:16.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.117+0000 7f7a5bc00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a541002d0 0x7f7a54110150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:16.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.117+0000 7f7a5bc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a54110750 con 0x7f7a540ff020 2024-04-16T05:48:16.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.117+0000 7f7a5bc00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a54110fa0 con 0x7f7a541002d0 2024-04-16T05:48:16.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.117+0000 7f7a59200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a540ff020 0x7f7a5410fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:16.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.118+0000 7f7a59200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a540ff020 0x7f7a5410fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40398/0 (socket says 172.21.15.27:40398) 2024-04-16T05:48:16.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.118+0000 7f7a59200700 1 -- 172.21.15.27:0/3009513212 learned_addr learned my addr 172.21.15.27:0/3009513212 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:48:16.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.118+0000 7f7a53e00700 1 --2- 172.21.15.27:0/3009513212 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a541002d0 0x7f7a54110150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:16.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.118+0000 7f7a59200700 1 -- 172.21.15.27:0/3009513212 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a541002d0 msgr2=0x7f7a54110150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:16.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.118+0000 7f7a59200700 1 --2- 172.21.15.27:0/3009513212 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a541002d0 0x7f7a54110150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:16.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.118+0000 7f7a59200700 1 -- 172.21.15.27:0/3009513212 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a48005040 con 0x7f7a540ff020 2024-04-16T05:48:16.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.118+0000 7f7a59200700 1 --2- 172.21.15.27:0/3009513212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a540ff020 0x7f7a5410fc10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7a4800b3b0 tx=0x7f7a48009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:16.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.119+0000 7f7a50c00700 1 -- 172.21.15.27:0/3009513212 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a4800e040 con 0x7f7a540ff020 2024-04-16T05:48:16.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.119+0000 7f7a50c00700 1 -- 172.21.15.27:0/3009513212 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a48004290 con 0x7f7a540ff020 2024-04-16T05:48:16.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.119+0000 7f7a5bc00700 1 -- 172.21.15.27:0/3009513212 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a54111220 con 0x7f7a540ff020 2024-04-16T05:48:16.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.119+0000 7f7a50c00700 1 -- 172.21.15.27:0/3009513212 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a4801c630 con 0x7f7a540ff020 2024-04-16T05:48:16.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.119+0000 7f7a5bc00700 1 -- 172.21.15.27:0/3009513212 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a54111760 con 0x7f7a540ff020 2024-04-16T05:48:16.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.120+0000 7f7a5bc00700 1 -- 172.21.15.27:0/3009513212 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a5405f5f0 con 0x7f7a540ff020 2024-04-16T05:48:16.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.121+0000 7f7a50c00700 1 -- 172.21.15.27:0/3009513212 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7a4800bbf0 con 0x7f7a540ff020 2024-04-16T05:48:16.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.123+0000 7f7a50c00700 1 --2- 172.21.15.27:0/3009513212 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7a3c06ec10 0x7f7a3c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:16.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.123+0000 7f7a50c00700 1 -- 172.21.15.27:0/3009513212 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f7a4808fe10 con 0x7f7a540ff020 2024-04-16T05:48:16.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.123+0000 7f7a53e00700 1 --2- 172.21.15.27:0/3009513212 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7a3c06ec10 0x7f7a3c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:16.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.124+0000 7f7a53e00700 1 --2- 172.21.15.27:0/3009513212 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7a3c06ec10 0x7f7a3c0710c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7a40004990 tx=0x7f7a40009ee0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:16.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.127+0000 7f7a50c00700 1 -- 172.21.15.27:0/3009513212 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7a4805c600 con 0x7f7a540ff020 2024-04-16T05:48:16.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.387+0000 7f7a5bc00700 1 -- 172.21.15.27:0/3009513212 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a54111a90 con 0x7f7a3c06ec10 2024-04-16T05:48:16.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.388+0000 7f7a50c00700 1 -- 172.21.15.27:0/3009513212 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f7a54111a90 con 0x7f7a3c06ec10 2024-04-16T05:48:16.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.391+0000 7f7a5bc00700 1 -- 172.21.15.27:0/3009513212 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7a3c06ec10 msgr2=0x7f7a3c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:16.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.391+0000 7f7a5bc00700 1 --2- 172.21.15.27:0/3009513212 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7a3c06ec10 0x7f7a3c0710c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7a40004990 tx=0x7f7a40009ee0 comp rx=0 tx=0).stop 2024-04-16T05:48:16.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.391+0000 7f7a5bc00700 1 -- 172.21.15.27:0/3009513212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a540ff020 msgr2=0x7f7a5410fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:16.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.391+0000 7f7a5bc00700 1 --2- 172.21.15.27:0/3009513212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a540ff020 0x7f7a5410fc10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7a4800b3b0 tx=0x7f7a48009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:16.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.392+0000 7f7a5bc00700 1 -- 172.21.15.27:0/3009513212 shutdown_connections 2024-04-16T05:48:16.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.392+0000 7f7a5bc00700 1 --2- 172.21.15.27:0/3009513212 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7a3c06ec10 0x7f7a3c0710c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:16.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.392+0000 7f7a5bc00700 1 --2- 172.21.15.27:0/3009513212 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a540ff020 0x7f7a5410fc10 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:16.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.392+0000 7f7a5bc00700 1 --2- 172.21.15.27:0/3009513212 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a541002d0 0x7f7a54110150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:16.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.392+0000 7f7a5bc00700 1 -- 172.21.15.27:0/3009513212 >> 172.21.15.27:0/3009513212 conn(0x7f7a540fa950 msgr2=0x7f7a540fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:16.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.392+0000 7f7a5bc00700 1 -- 172.21.15.27:0/3009513212 shutdown_connections 2024-04-16T05:48:16.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.392+0000 7f7a5bc00700 1 -- 172.21.15.27:0/3009513212 wait complete. 2024-04-16T05:48:16.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.543+0000 7f2d0e000700 1 -- 172.21.15.27:0/745115125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d081002d0 msgr2=0x7f2d08100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:16.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.543+0000 7f2d0e000700 1 --2- 172.21.15.27:0/745115125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d081002d0 0x7f2d08100740 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4004940 tx=0x7f2cf4009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:16.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.543+0000 7f2d0e000700 1 -- 172.21.15.27:0/745115125 shutdown_connections 2024-04-16T05:48:16.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.543+0000 7f2d0e000700 1 --2- 172.21.15.27:0/745115125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d081002d0 0x7f2d08100740 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:16.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.543+0000 7f2d0e000700 1 --2- 172.21.15.27:0/745115125 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d080ff020 0x7f2d080ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:16.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.543+0000 7f2d0e000700 1 -- 172.21.15.27:0/745115125 >> 172.21.15.27:0/745115125 conn(0x7f2d080fa950 msgr2=0x7f2d080fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:16.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.543+0000 7f2d0e000700 1 -- 172.21.15.27:0/745115125 shutdown_connections 2024-04-16T05:48:16.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.543+0000 7f2d0e000700 1 -- 172.21.15.27:0/745115125 wait complete. 2024-04-16T05:48:16.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.544+0000 7f2d0e000700 1 Processor -- start 2024-04-16T05:48:16.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.544+0000 7f2d0e000700 1 -- start start 2024-04-16T05:48:16.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.544+0000 7f2d0e000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d080ff020 0x7f2d0810fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:16.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.544+0000 7f2d0e000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d081002d0 0x7f2d081100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:16.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.545+0000 7f2d0e000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d081106b0 con 0x7f2d080ff020 2024-04-16T05:48:16.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.545+0000 7f2d0e000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d08110820 con 0x7f2d081002d0 2024-04-16T05:48:16.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.545+0000 7f2d07400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d080ff020 0x7f2d0810fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:16.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.545+0000 7f2d07400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d080ff020 0x7f2d0810fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40408/0 (socket says 172.21.15.27:40408) 2024-04-16T05:48:16.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.545+0000 7f2d07400700 1 -- 172.21.15.27:0/1803528790 learned_addr learned my addr 172.21.15.27:0/1803528790 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:48:16.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.545+0000 7f2d06a00700 1 --2- 172.21.15.27:0/1803528790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d081002d0 0x7f2d081100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:16.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.545+0000 7f2d07400700 1 -- 172.21.15.27:0/1803528790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d081002d0 msgr2=0x7f2d081100b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:16.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.545+0000 7f2d07400700 1 --2- 172.21.15.27:0/1803528790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d081002d0 0x7f2d081100b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:16.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.545+0000 7f2d07400700 1 -- 172.21.15.27:0/1803528790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf4005040 con 0x7f2d080ff020 2024-04-16T05:48:16.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.546+0000 7f2d07400700 1 --2- 172.21.15.27:0/1803528790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d080ff020 0x7f2d0810fb70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2cf8002a50 tx=0x7f2cf80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:16.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.546+0000 7f2cff400700 1 -- 172.21.15.27:0/1803528790 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cf8007cc0 con 0x7f2d080ff020 2024-04-16T05:48:16.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.546+0000 7f2cff400700 1 -- 172.21.15.27:0/1803528790 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2cf800f040 con 0x7f2d080ff020 2024-04-16T05:48:16.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.546+0000 7f2cff400700 1 -- 172.21.15.27:0/1803528790 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cf8009640 con 0x7f2d080ff020 2024-04-16T05:48:16.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.546+0000 7f2d0e000700 1 -- 172.21.15.27:0/1803528790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d08115280 con 0x7f2d080ff020 2024-04-16T05:48:16.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.546+0000 7f2d0e000700 1 -- 172.21.15.27:0/1803528790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d08115840 con 0x7f2d080ff020 2024-04-16T05:48:16.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.547+0000 7f2d0e000700 1 -- 172.21.15.27:0/1803528790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d0804c710 con 0x7f2d080ff020 2024-04-16T05:48:16.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.548+0000 7f2cff400700 1 -- 172.21.15.27:0/1803528790 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2cf80097a0 con 0x7f2d080ff020 2024-04-16T05:48:16.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.549+0000 7f2cff400700 1 --2- 172.21.15.27:0/1803528790 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2cec06ec00 0x7f2cec0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:16.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.549+0000 7f2cff400700 1 -- 172.21.15.27:0/1803528790 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f2cf808e6a0 con 0x7f2d080ff020 2024-04-16T05:48:16.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.550+0000 7f2d06a00700 1 --2- 172.21.15.27:0/1803528790 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2cec06ec00 0x7f2cec0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:16.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.550+0000 7f2d06a00700 1 --2- 172.21.15.27:0/1803528790 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2cec06ec00 0x7f2cec0710b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2cf400b3b0 tx=0x7f2cf4009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:16.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.554+0000 7f2cff400700 1 -- 172.21.15.27:0/1803528790 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2cf805af10 con 0x7f2d080ff020 2024-04-16T05:48:16.639 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:16 smithi027 ceph-mon[93635]: pgmap v100: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:16.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.798+0000 7f2d0e000700 1 -- 172.21.15.27:0/1803528790 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d081045f0 con 0x7f2cec06ec00 2024-04-16T05:48:16.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.814+0000 7f2cff400700 1 -- 172.21.15.27:0/1803528790 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f2d081045f0 con 0x7f2cec06ec00 2024-04-16T05:48:16.814 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:48:16.814 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (4m) 5s ago 15m 13.0M - 0.25.0 c8568f914cd2 4ed63a8a6b33 2024-04-16T05:48:16.814 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (84s) 5s ago 15m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 4aa1f0006685 2024-04-16T05:48:16.814 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (76s) 71s ago 14m 6656k - 19.0.0-2884-g218a6914 a4e4811815e4 1c1912507d56 2024-04-16T05:48:16.815 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (4m) 5s ago 14m 75.4M - 9.4.7 954c08fa6188 f7ee2a1afd12 2024-04-16T05:48:16.815 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (7m) 5s ago 7m 23.7M - 17.2.7-847-g45317f95 b333581ccdf5 0824e2621be5 2024-04-16T05:48:16.815 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (8m) 5s ago 7m 13.8M - 17.2.7-847-g45317f95 b333581ccdf5 b332a72cc594 2024-04-16T05:48:16.815 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (7m) 71s ago 7m 13.6M - 17.2.7-847-g45317f95 b333581ccdf5 f5e36815ddfa 2024-04-16T05:48:16.815 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (7m) 71s ago 7m 13.5M - 17.2.7-847-g45317f95 b333581ccdf5 755a73c65a7f 2024-04-16T05:48:16.815 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:8443,9283,8765 running (4m) 5s ago 17m 500M - 19.0.0-2884-g218a6914 a4e4811815e4 b69a0b08e12b 2024-04-16T05:48:16.815 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283,8765 running (3m) 71s ago 13m 451M - 19.0.0-2884-g218a6914 a4e4811815e4 32e27bcd7e42 2024-04-16T05:48:16.815 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (2m) 5s ago 17m 44.5M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 299ca337e4ca 2024-04-16T05:48:16.815 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (2m) 71s ago 13m 31.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5db5f3037ed0 2024-04-16T05:48:16.815 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (4m) 5s ago 15m 7612k - 1.5.0 0da6a335fe13 5e29e29868b0 2024-04-16T05:48:16.815 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (4m) 71s ago 14m 8032k - 1.5.0 0da6a335fe13 a360922bc621 2024-04-16T05:48:16.815 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (45s) 5s ago 12m 41.7M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 cef03b67bc5a 2024-04-16T05:48:16.815 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (13s) 5s ago 12m 11.1M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3a6ab1fc6e79 2024-04-16T05:48:16.815 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (11m) 5s ago 11m 50.4M 1557M 17.2.7-847-g45317f95 b333581ccdf5 6212f6a18fa4 2024-04-16T05:48:16.815 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (11m) 71s ago 10m 46.3M 2581M 17.2.7-847-g45317f95 b333581ccdf5 d6724ad05f9f 2024-04-16T05:48:16.815 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (10m) 71s ago 10m 45.0M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7a59908a3e10 2024-04-16T05:48:16.816 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (9m) 71s ago 9m 44.2M 2581M 17.2.7-847-g45317f95 b333581ccdf5 8e4b60163907 2024-04-16T05:48:16.816 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (3m) 5s ago 14m 55.4M - 2.43.0 a07b618ecd1d e70f55ba5ec0 2024-04-16T05:48:16.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.817+0000 7f2d0e000700 1 -- 172.21.15.27:0/1803528790 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2cec06ec00 msgr2=0x7f2cec0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:16.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.817+0000 7f2d0e000700 1 --2- 172.21.15.27:0/1803528790 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2cec06ec00 0x7f2cec0710b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2cf400b3b0 tx=0x7f2cf4009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:16.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.817+0000 7f2d0e000700 1 -- 172.21.15.27:0/1803528790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d080ff020 msgr2=0x7f2d0810fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.817+0000 7f2d0e000700 1 --2- 172.21.15.27:0/1803528790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d080ff020 0x7f2d0810fb70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2cf8002a50 tx=0x7f2cf80046b0 comp rx=0 tx=0).stop 2024-04-16T05:48:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.818+0000 7f2d0e000700 1 -- 172.21.15.27:0/1803528790 shutdown_connections 2024-04-16T05:48:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.818+0000 7f2d0e000700 1 --2- 172.21.15.27:0/1803528790 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2cec06ec00 0x7f2cec0710b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.818+0000 7f2d0e000700 1 --2- 172.21.15.27:0/1803528790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d080ff020 0x7f2d0810fb70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.818+0000 7f2d0e000700 1 --2- 172.21.15.27:0/1803528790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d081002d0 0x7f2d081100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.818+0000 7f2d0e000700 1 -- 172.21.15.27:0/1803528790 >> 172.21.15.27:0/1803528790 conn(0x7f2d080fa950 msgr2=0x7f2d080fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.818+0000 7f2d0e000700 1 -- 172.21.15.27:0/1803528790 shutdown_connections 2024-04-16T05:48:16.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.818+0000 7f2d0e000700 1 -- 172.21.15.27:0/1803528790 wait complete. 2024-04-16T05:48:16.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.970+0000 7f0831000700 1 -- 172.21.15.27:0/2255376452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f082c0662f0 msgr2=0x7f082c0ff800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:16.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.970+0000 7f0831000700 1 --2- 172.21.15.27:0/2255376452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f082c0662f0 0x7f082c0ff800 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f081c004940 tx=0x7f081c009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:16.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.970+0000 7f0831000700 1 -- 172.21.15.27:0/2255376452 shutdown_connections 2024-04-16T05:48:16.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.970+0000 7f0831000700 1 --2- 172.21.15.27:0/2255376452 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f082c0ffd40 0x7f082c102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:16.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.970+0000 7f0831000700 1 --2- 172.21.15.27:0/2255376452 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f082c0662f0 0x7f082c0ff800 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:16.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.970+0000 7f0831000700 1 -- 172.21.15.27:0/2255376452 >> 172.21.15.27:0/2255376452 conn(0x7f082c0f7270 msgr2=0x7f082c0f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:16.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.971+0000 7f0831000700 1 -- 172.21.15.27:0/2255376452 shutdown_connections 2024-04-16T05:48:16.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.971+0000 7f0831000700 1 -- 172.21.15.27:0/2255376452 wait complete. 2024-04-16T05:48:16.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.972+0000 7f0831000700 1 Processor -- start 2024-04-16T05:48:16.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.972+0000 7f0831000700 1 -- start start 2024-04-16T05:48:16.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.973+0000 7f0831000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f082c0662f0 0x7f082c0fd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:16.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.973+0000 7f0831000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f082c0ffd40 0x7f082c0fb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:16.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.973+0000 7f0831000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082c0fbcb0 con 0x7f082c0ffd40 2024-04-16T05:48:16.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.973+0000 7f0831000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082c0fbe20 con 0x7f082c0662f0 2024-04-16T05:48:16.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.973+0000 7f082aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f082c0ffd40 0x7f082c0fb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:16.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.973+0000 7f082aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f082c0ffd40 0x7f082c0fb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40434/0 (socket says 172.21.15.27:40434) 2024-04-16T05:48:16.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.973+0000 7f082aa00700 1 -- 172.21.15.27:0/2519729000 learned_addr learned my addr 172.21.15.27:0/2519729000 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:48:16.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.973+0000 7f082b400700 1 --2- 172.21.15.27:0/2519729000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f082c0662f0 0x7f082c0fd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:16.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.974+0000 7f082aa00700 1 -- 172.21.15.27:0/2519729000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f082c0662f0 msgr2=0x7f082c0fd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:16.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.974+0000 7f082aa00700 1 --2- 172.21.15.27:0/2519729000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f082c0662f0 0x7f082c0fd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:16.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.974+0000 7f082aa00700 1 -- 172.21.15.27:0/2519729000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081c005040 con 0x7f082c0ffd40 2024-04-16T05:48:16.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.974+0000 7f082aa00700 1 --2- 172.21.15.27:0/2519729000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f082c0ffd40 0x7f082c0fb700 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f081800c8c0 tx=0x7f08180044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:16.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.975+0000 7f0823400700 1 -- 172.21.15.27:0/2519729000 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f081800a760 con 0x7f082c0ffd40 2024-04-16T05:48:16.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.975+0000 7f0823400700 1 -- 172.21.15.27:0/2519729000 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f081800ad90 con 0x7f082c0ffd40 2024-04-16T05:48:16.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.975+0000 7f0831000700 1 -- 172.21.15.27:0/2519729000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f082c0fc100 con 0x7f082c0ffd40 2024-04-16T05:48:16.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.975+0000 7f0823400700 1 -- 172.21.15.27:0/2519729000 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08180113a0 con 0x7f082c0ffd40 2024-04-16T05:48:16.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.975+0000 7f0831000700 1 -- 172.21.15.27:0/2519729000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f082c1158d0 con 0x7f082c0ffd40 2024-04-16T05:48:16.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.976+0000 7f0831000700 1 -- 172.21.15.27:0/2519729000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f082c05f5f0 con 0x7f082c0ffd40 2024-04-16T05:48:16.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.978+0000 7f0823400700 1 -- 172.21.15.27:0/2519729000 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0818012400 con 0x7f082c0ffd40 2024-04-16T05:48:16.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.979+0000 7f0823400700 1 --2- 172.21.15.27:0/2519729000 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f081006ec00 0x7f08100710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:16.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.979+0000 7f0823400700 1 -- 172.21.15.27:0/2519729000 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f081808ea30 con 0x7f082c0ffd40 2024-04-16T05:48:16.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.979+0000 7f082b400700 1 --2- 172.21.15.27:0/2519729000 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f081006ec00 0x7f08100710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:16.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.980+0000 7f082b400700 1 --2- 172.21.15.27:0/2519729000 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f081006ec00 0x7f08100710b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f081c004890 tx=0x7f081c009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:16.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:16.988+0000 7f0823400700 1 -- 172.21.15.27:0/2519729000 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f08180576f0 con 0x7f082c0ffd40 2024-04-16T05:48:17.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:16 smithi122 ceph-mon[80930]: pgmap v100: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:17.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.298+0000 7f0831000700 1 -- 172.21.15.27:0/2519729000 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f082c04bed0 con 0x7f082c0ffd40 2024-04-16T05:48:17.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.300+0000 7f0823400700 1 -- 172.21.15.27:0/2519729000 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7f081805adb0 con 0x7f082c0ffd40 2024-04-16T05:48:17.299 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:48:17.299 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:48:17.299 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:48:17.299 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:48:17.299 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:48:17.300 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:48:17.300 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:48:17.300 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:48:17.300 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4, 2024-04-16T05:48:17.300 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:48:17.300 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:48:17.300 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:48:17.300 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T05:48:17.300 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:48:17.300 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:48:17.300 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 8, 2024-04-16T05:48:17.300 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T05:48:17.300 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:48:17.300 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:48:17.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.303+0000 7f0831000700 1 -- 172.21.15.27:0/2519729000 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f081006ec00 msgr2=0x7f08100710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:17.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.303+0000 7f0831000700 1 --2- 172.21.15.27:0/2519729000 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f081006ec00 0x7f08100710b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f081c004890 tx=0x7f081c009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:17.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.303+0000 7f0831000700 1 -- 172.21.15.27:0/2519729000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f082c0ffd40 msgr2=0x7f082c0fb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:17.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.303+0000 7f0831000700 1 --2- 172.21.15.27:0/2519729000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f082c0ffd40 0x7f082c0fb700 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f081800c8c0 tx=0x7f08180044c0 comp rx=0 tx=0).stop 2024-04-16T05:48:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.303+0000 7f0831000700 1 -- 172.21.15.27:0/2519729000 shutdown_connections 2024-04-16T05:48:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.303+0000 7f0831000700 1 --2- 172.21.15.27:0/2519729000 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f081006ec00 0x7f08100710b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.303+0000 7f0831000700 1 --2- 172.21.15.27:0/2519729000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f082c0662f0 0x7f082c0fd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.303+0000 7f0831000700 1 --2- 172.21.15.27:0/2519729000 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f082c0ffd40 0x7f082c0fb700 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.303+0000 7f0831000700 1 -- 172.21.15.27:0/2519729000 >> 172.21.15.27:0/2519729000 conn(0x7f082c0f7270 msgr2=0x7f082c0f7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.304+0000 7f0831000700 1 -- 172.21.15.27:0/2519729000 shutdown_connections 2024-04-16T05:48:17.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.304+0000 7f0831000700 1 -- 172.21.15.27:0/2519729000 wait complete. 2024-04-16T05:48:17.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.450+0000 7ff10e600700 1 -- 172.21.15.27:0/2114471025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1080feff0 msgr2=0x7ff1080ff400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:17.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.450+0000 7ff10e600700 1 --2- 172.21.15.27:0/2114471025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1080feff0 0x7ff1080ff400 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8004940 tx=0x7ff0f8009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:17.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.450+0000 7ff10e600700 1 -- 172.21.15.27:0/2114471025 shutdown_connections 2024-04-16T05:48:17.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.450+0000 7ff10e600700 1 --2- 172.21.15.27:0/2114471025 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1081002a0 0x7ff108100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:17.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.450+0000 7ff10e600700 1 --2- 172.21.15.27:0/2114471025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1080feff0 0x7ff1080ff400 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:17.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.450+0000 7ff10e600700 1 -- 172.21.15.27:0/2114471025 >> 172.21.15.27:0/2114471025 conn(0x7ff1080fa790 msgr2=0x7ff1080fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:17.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.450+0000 7ff10e600700 1 -- 172.21.15.27:0/2114471025 shutdown_connections 2024-04-16T05:48:17.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.450+0000 7ff10e600700 1 -- 172.21.15.27:0/2114471025 wait complete. 2024-04-16T05:48:17.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.452+0000 7ff10e600700 1 Processor -- start 2024-04-16T05:48:17.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.452+0000 7ff10e600700 1 -- start start 2024-04-16T05:48:17.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.452+0000 7ff10e600700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1080feff0 0x7ff10810fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:17.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.452+0000 7ff10e600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1081002a0 0x7ff108110050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:17.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.452+0000 7ff10e600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff108110650 con 0x7ff1081002a0 2024-04-16T05:48:17.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.452+0000 7ff10e600700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1081107c0 con 0x7ff1080feff0 2024-04-16T05:48:17.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.453+0000 7ff106a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1081002a0 0x7ff108110050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:17.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.453+0000 7ff106a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1081002a0 0x7ff108110050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40458/0 (socket says 172.21.15.27:40458) 2024-04-16T05:48:17.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.453+0000 7ff106a00700 1 -- 172.21.15.27:0/1301324658 learned_addr learned my addr 172.21.15.27:0/1301324658 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:48:17.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.453+0000 7ff107400700 1 --2- 172.21.15.27:0/1301324658 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1080feff0 0x7ff10810fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:17.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.453+0000 7ff106a00700 1 -- 172.21.15.27:0/1301324658 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1080feff0 msgr2=0x7ff10810fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:17.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.453+0000 7ff106a00700 1 --2- 172.21.15.27:0/1301324658 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1080feff0 0x7ff10810fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:17.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.453+0000 7ff106a00700 1 -- 172.21.15.27:0/1301324658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f8005040 con 0x7ff1081002a0 2024-04-16T05:48:17.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.453+0000 7ff106a00700 1 --2- 172.21.15.27:0/1301324658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1081002a0 0x7ff108110050 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400c8c0 tx=0x7ff0f40044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:17.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.454+0000 7ff0ff400700 1 -- 172.21.15.27:0/1301324658 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0f400a980 con 0x7ff1081002a0 2024-04-16T05:48:17.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.454+0000 7ff0ff400700 1 -- 172.21.15.27:0/1301324658 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff0f400ce60 con 0x7ff1081002a0 2024-04-16T05:48:17.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.454+0000 7ff10e600700 1 -- 172.21.15.27:0/1301324658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff108115220 con 0x7ff1081002a0 2024-04-16T05:48:17.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.454+0000 7ff0ff400700 1 -- 172.21.15.27:0/1301324658 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0f40115e0 con 0x7ff1081002a0 2024-04-16T05:48:17.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.454+0000 7ff10e600700 1 -- 172.21.15.27:0/1301324658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1081157e0 con 0x7ff1081002a0 2024-04-16T05:48:17.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.455+0000 7ff10e600700 1 -- 172.21.15.27:0/1301324658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff10804c710 con 0x7ff1081002a0 2024-04-16T05:48:17.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.456+0000 7ff0ff400700 1 -- 172.21.15.27:0/1301324658 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff0f4012400 con 0x7ff1081002a0 2024-04-16T05:48:17.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.457+0000 7ff0ff400700 1 --2- 172.21.15.27:0/1301324658 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff0ec06ec00 0x7ff0ec0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:17.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.457+0000 7ff0ff400700 1 -- 172.21.15.27:0/1301324658 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6736+0+0 (secure 0 0 0) 0x7ff0f408e940 con 0x7ff1081002a0 2024-04-16T05:48:17.456 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.458+0000 7ff107400700 1 --2- 172.21.15.27:0/1301324658 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff0ec06ec00 0x7ff0ec0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:17.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.458+0000 7ff107400700 1 --2- 172.21.15.27:0/1301324658 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff0ec06ec00 0x7ff0ec0710b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8004f90 tx=0x7ff0f8009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:17.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.462+0000 7ff0ff400700 1 -- 172.21.15.27:0/1301324658 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff0f405b270 con 0x7ff1081002a0 2024-04-16T05:48:17.807 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 240, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110054 2024-04-16T05:48:17.808 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 240, success, outsize: 120 2024-04-16T05:48:17.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.815+0000 7ff10e600700 1 -- 172.21.15.27:0/1301324658 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff108107ae0 con 0x7ff1081002a0 2024-04-16T05:48:17.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.816+0000 7ff0ff400700 1 -- 172.21.15.27:0/1301324658 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1880 (secure 0 0 0) 0x7ff0f40126b0 con 0x7ff1081002a0 2024-04-16T05:48:17.815 INFO:teuthology.orchestra.run.smithi027.stdout:e16 2024-04-16T05:48:17.815 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:48:17.815 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:48:17.815 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:48:17.815 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:48:17.815 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:48:17.815 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:48:17.815 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 15 2024-04-16T05:48:17.815 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:48:17.816 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:48:17.816 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:40:32.701649+0000 2024-04-16T05:48:17.816 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:48:17.816 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:48:17.816 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:48:17.816 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:48:17.816 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:48:17.816 INFO:teuthology.orchestra.run.smithi027.stdout:max_xattr_size 65536 2024-04-16T05:48:17.816 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:48:17.816 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:48:17.816 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 46 2024-04-16T05:48:17.816 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:48:17.816 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:48:17.816 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:48:17.816 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=14462} 2024-04-16T05:48:17.817 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:48:17.817 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:48:17.817 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:48:17.817 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:48:17.817 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:48:17.817 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:48:17.817 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:48:17.817 INFO:teuthology.orchestra.run.smithi027.stdout:bal_rank_mask -1 2024-04-16T05:48:17.817 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:48:17.817 INFO:teuthology.orchestra.run.smithi027.stdout:qdb_cluster leader: 0 members: 2024-04-16T05:48:17.817 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:48:17.817 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:48:17.817 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:48:17.817 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:48:17.817 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:48:17.818 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{-1:14474} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2065690722,v1:172.21.15.27:6827/2065690722] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:48:17.818 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:24263} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:48:17.818 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:24273} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:48:17.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.818+0000 7ff10e600700 1 -- 172.21.15.27:0/1301324658 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff0ec06ec00 msgr2=0x7ff0ec0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:17.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.818+0000 7ff10e600700 1 --2- 172.21.15.27:0/1301324658 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff0ec06ec00 0x7ff0ec0710b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8004f90 tx=0x7ff0f8009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:17.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.818+0000 7ff10e600700 1 -- 172.21.15.27:0/1301324658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1081002a0 msgr2=0x7ff108110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:17.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.818+0000 7ff10e600700 1 --2- 172.21.15.27:0/1301324658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1081002a0 0x7ff108110050 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400c8c0 tx=0x7ff0f40044c0 comp rx=0 tx=0).stop 2024-04-16T05:48:17.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.818+0000 7ff10e600700 1 -- 172.21.15.27:0/1301324658 shutdown_connections 2024-04-16T05:48:17.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.818+0000 7ff10e600700 1 --2- 172.21.15.27:0/1301324658 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff0ec06ec00 0x7ff0ec0710b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:17.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.818+0000 7ff10e600700 1 --2- 172.21.15.27:0/1301324658 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1080feff0 0x7ff10810fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:17.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.818+0000 7ff10e600700 1 --2- 172.21.15.27:0/1301324658 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1081002a0 0x7ff108110050 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:17.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.818+0000 7ff10e600700 1 -- 172.21.15.27:0/1301324658 >> 172.21.15.27:0/1301324658 conn(0x7ff1080fa790 msgr2=0x7ff1080fbe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:17.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.819+0000 7ff10e600700 1 -- 172.21.15.27:0/1301324658 shutdown_connections 2024-04-16T05:48:17.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.819+0000 7ff10e600700 1 -- 172.21.15.27:0/1301324658 wait complete. 2024-04-16T05:48:17.820 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2024-04-16T05:48:17.829 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 242, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110154 2024-04-16T05:48:17.829 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 242, success, outsize: 120 2024-04-16T05:48:17.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 244, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110256 2024-04-16T05:48:17.847 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 244, success, outsize: 120 2024-04-16T05:48:17.866 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 246, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110363 2024-04-16T05:48:17.866 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 246, success, outsize: 120 2024-04-16T05:48:17.886 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 248, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110467 2024-04-16T05:48:17.886 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 248, success, outsize: 120 2024-04-16T05:48:17.904 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 250, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110565 2024-04-16T05:48:17.904 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 250, success, outsize: 120 2024-04-16T05:48:17.926 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 252, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110666 2024-04-16T05:48:17.926 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 252, success, outsize: 120 2024-04-16T05:48:17.950 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 254, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110774 2024-04-16T05:48:17.950 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 254, success, outsize: 120 2024-04-16T05:48:17.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 256, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110881 2024-04-16T05:48:17.970 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 256, success, outsize: 120 2024-04-16T05:48:17.973 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:17 smithi027 ceph-mon[93635]: from='client.34276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:48:17.974 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:17 smithi027 ceph-mon[93635]: from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:48:17.974 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:17 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/2519729000' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:48:17.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.974+0000 7f72ce000700 1 -- 172.21.15.27:0/4270720688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72c8066270 msgr2=0x7f72c8101eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:17.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.974+0000 7f72ce000700 1 --2- 172.21.15.27:0/4270720688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72c8066270 0x7f72c8101eb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f72b8004940 tx=0x7f72b8009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:17.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.975+0000 7f72ce000700 1 -- 172.21.15.27:0/4270720688 shutdown_connections 2024-04-16T05:48:17.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.975+0000 7f72ce000700 1 --2- 172.21.15.27:0/4270720688 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f72c81023f0 0x7f72c81047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:17.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.975+0000 7f72ce000700 1 --2- 172.21.15.27:0/4270720688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72c8066270 0x7f72c8101eb0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:17.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.975+0000 7f72ce000700 1 -- 172.21.15.27:0/4270720688 >> 172.21.15.27:0/4270720688 conn(0x7f72c80f74c0 msgr2=0x7f72c80f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:17.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.975+0000 7f72ce000700 1 -- 172.21.15.27:0/4270720688 shutdown_connections 2024-04-16T05:48:17.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.975+0000 7f72ce000700 1 -- 172.21.15.27:0/4270720688 wait complete. 2024-04-16T05:48:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.976+0000 7f72ce000700 1 Processor -- start 2024-04-16T05:48:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.976+0000 7f72ce000700 1 -- start start 2024-04-16T05:48:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.976+0000 7f72ce000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72c8066270 0x7f72c810fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.977+0000 7f72ce000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f72c81023f0 0x7f72c81102b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.977+0000 7f72ce000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c81108b0 con 0x7f72c8066270 2024-04-16T05:48:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.977+0000 7f72ce000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c8110ff0 con 0x7f72c81023f0 2024-04-16T05:48:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.977+0000 7f72c7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72c8066270 0x7f72c810fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.977+0000 7f72c7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72c8066270 0x7f72c810fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40476/0 (socket says 172.21.15.27:40476) 2024-04-16T05:48:17.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.977+0000 7f72c7400700 1 -- 172.21.15.27:0/2564993145 learned_addr learned my addr 172.21.15.27:0/2564993145 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:48:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.977+0000 7f72c6a00700 1 --2- 172.21.15.27:0/2564993145 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f72c81023f0 0x7f72c81102b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.977+0000 7f72c7400700 1 -- 172.21.15.27:0/2564993145 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f72c81023f0 msgr2=0x7f72c81102b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.977+0000 7f72c7400700 1 --2- 172.21.15.27:0/2564993145 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f72c81023f0 0x7f72c81102b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:17.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.978+0000 7f72c7400700 1 -- 172.21.15.27:0/2564993145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b8005040 con 0x7f72c8066270 2024-04-16T05:48:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.978+0000 7f72c7400700 1 --2- 172.21.15.27:0/2564993145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72c8066270 0x7f72c810fd70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f72b8009150 tx=0x7f72b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.978+0000 7f72bf400700 1 -- 172.21.15.27:0/2564993145 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72b800e040 con 0x7f72c8066270 2024-04-16T05:48:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.978+0000 7f72ce000700 1 -- 172.21.15.27:0/2564993145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72c8111270 con 0x7f72c8066270 2024-04-16T05:48:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.978+0000 7f72bf400700 1 -- 172.21.15.27:0/2564993145 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f72b800bb90 con 0x7f72c8066270 2024-04-16T05:48:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.978+0000 7f72bf400700 1 -- 172.21.15.27:0/2564993145 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f72b801c3f0 con 0x7f72c8066270 2024-04-16T05:48:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.979+0000 7f72ce000700 1 -- 172.21.15.27:0/2564993145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72c8111830 con 0x7f72c8066270 2024-04-16T05:48:17.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.980+0000 7f72ce000700 1 -- 172.21.15.27:0/2564993145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72c805f5f0 con 0x7f72c8066270 2024-04-16T05:48:17.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.981+0000 7f72bf400700 1 -- 172.21.15.27:0/2564993145 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f72b800bd00 con 0x7f72c8066270 2024-04-16T05:48:17.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.982+0000 7f72bf400700 1 --2- 172.21.15.27:0/2564993145 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f72ac06ec00 0x7f72ac0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:17.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.982+0000 7f72bf400700 1 -- 172.21.15.27:0/2564993145 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f72b808fca0 con 0x7f72c8066270 2024-04-16T05:48:17.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.983+0000 7f72c6a00700 1 --2- 172.21.15.27:0/2564993145 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f72ac06ec00 0x7f72ac0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:17.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.984+0000 7f72c6a00700 1 --2- 172.21.15.27:0/2564993145 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f72ac06ec00 0x7f72ac0710b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f72b4007fd0 tx=0x7f72b4009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:17.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:17.987+0000 7f72bf400700 1 -- 172.21.15.27:0/2564993145 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f72b8094070 con 0x7f72c8066270 2024-04-16T05:48:17.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 258, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110995 2024-04-16T05:48:17.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 258, success, outsize: 120 2024-04-16T05:48:18.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 260, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111096 2024-04-16T05:48:18.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 260, success, outsize: 120 2024-04-16T05:48:18.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:17 smithi122 ceph-mon[80930]: from='client.34276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:48:18.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:17 smithi122 ceph-mon[80930]: from='client.34280 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:48:18.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:17 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/2519729000' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:48:18.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 262, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111203 2024-04-16T05:48:18.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 262, success, outsize: 120 2024-04-16T05:48:18.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 264, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111307 2024-04-16T05:48:18.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 264, success, outsize: 120 2024-04-16T05:48:18.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 266, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111405 2024-04-16T05:48:18.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 266, success, outsize: 120 2024-04-16T05:48:18.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 268, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111506 2024-04-16T05:48:18.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 268, success, outsize: 120 2024-04-16T05:48:18.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 270, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111613 2024-04-16T05:48:18.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 270, success, outsize: 120 2024-04-16T05:48:18.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 272, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111717 2024-04-16T05:48:18.113 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 272, success, outsize: 120 2024-04-16T05:48:18.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 274, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111815 2024-04-16T05:48:18.129 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 274, success, outsize: 120 2024-04-16T05:48:18.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 276, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111916 2024-04-16T05:48:18.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 276, success, outsize: 120 2024-04-16T05:48:18.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 278, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112023 2024-04-16T05:48:18.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 278, success, outsize: 120 2024-04-16T05:48:18.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 280, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112127 2024-04-16T05:48:18.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 280, success, outsize: 120 2024-04-16T05:48:18.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 282, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112225 2024-04-16T05:48:18.200 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 282, success, outsize: 120 2024-04-16T05:48:18.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 284, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112326 2024-04-16T05:48:18.218 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 284, success, outsize: 120 2024-04-16T05:48:18.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.240+0000 7f72ce000700 1 -- 172.21.15.27:0/2564993145 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f72c8111c70 con 0x7f72ac06ec00 2024-04-16T05:48:18.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.241+0000 7f72bf400700 1 -- 172.21.15.27:0/2564993145 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f72c8111c70 con 0x7f72ac06ec00 2024-04-16T05:48:18.240 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:48:18.240 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T05:48:18.240 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:48:18.240 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:48:18.241 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [ 2024-04-16T05:48:18.241 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr", 2024-04-16T05:48:18.241 INFO:teuthology.orchestra.run.smithi027.stdout: "mon", 2024-04-16T05:48:18.241 INFO:teuthology.orchestra.run.smithi027.stdout: "crash" 2024-04-16T05:48:18.241 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-16T05:48:18.241 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "8/21 daemons upgraded", 2024-04-16T05:48:18.241 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Currently upgrading osd daemons", 2024-04-16T05:48:18.241 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:48:18.241 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:48:18.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 286, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112434 2024-04-16T05:48:18.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 286, success, outsize: 120 2024-04-16T05:48:18.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.244+0000 7f72ce000700 1 -- 172.21.15.27:0/2564993145 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f72ac06ec00 msgr2=0x7f72ac0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:18.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.244+0000 7f72ce000700 1 --2- 172.21.15.27:0/2564993145 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f72ac06ec00 0x7f72ac0710b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f72b4007fd0 tx=0x7f72b4009040 comp rx=0 tx=0).stop 2024-04-16T05:48:18.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.244+0000 7f72ce000700 1 -- 172.21.15.27:0/2564993145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72c8066270 msgr2=0x7f72c810fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:18.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.244+0000 7f72ce000700 1 --2- 172.21.15.27:0/2564993145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72c8066270 0x7f72c810fd70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f72b8009150 tx=0x7f72b8009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:18.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.244+0000 7f72ce000700 1 -- 172.21.15.27:0/2564993145 shutdown_connections 2024-04-16T05:48:18.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.244+0000 7f72ce000700 1 --2- 172.21.15.27:0/2564993145 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f72ac06ec00 0x7f72ac0710b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:18.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.244+0000 7f72ce000700 1 --2- 172.21.15.27:0/2564993145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72c8066270 0x7f72c810fd70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.244+0000 7f72ce000700 1 --2- 172.21.15.27:0/2564993145 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f72c81023f0 0x7f72c81102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.244+0000 7f72ce000700 1 -- 172.21.15.27:0/2564993145 >> 172.21.15.27:0/2564993145 conn(0x7f72c80f74c0 msgr2=0x7f72c80f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.244+0000 7f72ce000700 1 -- 172.21.15.27:0/2564993145 shutdown_connections 2024-04-16T05:48:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.245+0000 7f72ce000700 1 -- 172.21.15.27:0/2564993145 wait complete. 2024-04-16T05:48:18.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 288, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112540 2024-04-16T05:48:18.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 288, success, outsize: 120 2024-04-16T05:48:18.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 290, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112638 2024-04-16T05:48:18.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 290, success, outsize: 120 2024-04-16T05:48:18.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 292, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112739 2024-04-16T05:48:18.306 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 292, success, outsize: 120 2024-04-16T05:48:18.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 294, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112846 2024-04-16T05:48:18.324 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 294, success, outsize: 120 2024-04-16T05:48:18.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.398+0000 7ff744e00700 1 -- 172.21.15.27:0/2319752666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7400fce40 msgr2=0x7ff7400fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:18.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.398+0000 7ff744e00700 1 --2- 172.21.15.27:0/2319752666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7400fce40 0x7ff7400fd250 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff730004940 tx=0x7ff730009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:18.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.399+0000 7ff744e00700 1 -- 172.21.15.27:0/2319752666 shutdown_connections 2024-04-16T05:48:18.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.399+0000 7ff744e00700 1 --2- 172.21.15.27:0/2319752666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7400fe040 0x7ff7400fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:18.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.399+0000 7ff744e00700 1 --2- 172.21.15.27:0/2319752666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7400fce40 0x7ff7400fd250 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:18.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.399+0000 7ff744e00700 1 -- 172.21.15.27:0/2319752666 >> 172.21.15.27:0/2319752666 conn(0x7ff7400f8530 msgr2=0x7ff7400fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:18.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.399+0000 7ff744e00700 1 -- 172.21.15.27:0/2319752666 shutdown_connections 2024-04-16T05:48:18.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.399+0000 7ff744e00700 1 -- 172.21.15.27:0/2319752666 wait complete. 2024-04-16T05:48:18.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.400+0000 7ff744e00700 1 Processor -- start 2024-04-16T05:48:18.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.400+0000 7ff744e00700 1 -- start start 2024-04-16T05:48:18.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.400+0000 7ff744e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7400fce40 0x7ff740071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:18.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.400+0000 7ff744e00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7400fe040 0x7ff740070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:18.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.400+0000 7ff744e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7400708c0 con 0x7ff7400fce40 2024-04-16T05:48:18.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.400+0000 7ff744e00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff740070a30 con 0x7ff7400fe040 2024-04-16T05:48:18.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.401+0000 7ff73de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7400fce40 0x7ff740071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:18.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.401+0000 7ff73de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7400fce40 0x7ff740071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40502/0 (socket says 172.21.15.27:40502) 2024-04-16T05:48:18.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.401+0000 7ff73de00700 1 -- 172.21.15.27:0/631486386 learned_addr learned my addr 172.21.15.27:0/631486386 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:48:18.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.401+0000 7ff73d400700 1 --2- 172.21.15.27:0/631486386 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7400fe040 0x7ff740070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.401+0000 7ff73de00700 1 -- 172.21.15.27:0/631486386 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7400fe040 msgr2=0x7ff740070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.401+0000 7ff73de00700 1 --2- 172.21.15.27:0/631486386 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7400fe040 0x7ff740070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.401+0000 7ff73de00700 1 -- 172.21.15.27:0/631486386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff730005040 con 0x7ff7400fce40 2024-04-16T05:48:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.401+0000 7ff73de00700 1 --2- 172.21.15.27:0/631486386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7400fce40 0x7ff740071e40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff73000b970 tx=0x7ff730009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.402+0000 7ff736000700 1 -- 172.21.15.27:0/631486386 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff73000e040 con 0x7ff7400fce40 2024-04-16T05:48:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.402+0000 7ff736000700 1 -- 172.21.15.27:0/631486386 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff730004290 con 0x7ff7400fce40 2024-04-16T05:48:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.402+0000 7ff744e00700 1 -- 172.21.15.27:0/631486386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff740070cb0 con 0x7ff7400fce40 2024-04-16T05:48:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.402+0000 7ff736000700 1 -- 172.21.15.27:0/631486386 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff73001c630 con 0x7ff7400fce40 2024-04-16T05:48:18.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.402+0000 7ff744e00700 1 -- 172.21.15.27:0/631486386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff740115ac0 con 0x7ff7400fce40 2024-04-16T05:48:18.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.403+0000 7ff744e00700 1 -- 172.21.15.27:0/631486386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff74005f5f0 con 0x7ff7400fce40 2024-04-16T05:48:18.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.404+0000 7ff736000700 1 -- 172.21.15.27:0/631486386 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff73000bbf0 con 0x7ff7400fce40 2024-04-16T05:48:18.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.405+0000 7ff736000700 1 --2- 172.21.15.27:0/631486386 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff72406ec10 0x7ff7240710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:18.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.405+0000 7ff736000700 1 -- 172.21.15.27:0/631486386 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6736+0+0 (secure 0 0 0) 0x7ff73008fc90 con 0x7ff7400fce40 2024-04-16T05:48:18.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.406+0000 7ff73d400700 1 --2- 172.21.15.27:0/631486386 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff72406ec10 0x7ff7240710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:18.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.407+0000 7ff73d400700 1 --2- 172.21.15.27:0/631486386 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff72406ec10 0x7ff7240710c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff72c004a30 tx=0x7ff72c007ab0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:18.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.410+0000 7ff736000700 1 -- 172.21.15.27:0/631486386 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff7300589d0 con 0x7ff7400fce40 2024-04-16T05:48:18.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.737+0000 7ff744e00700 1 -- 172.21.15.27:0/631486386 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff74004bed0 con 0x7ff7400fce40 2024-04-16T05:48:18.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.738+0000 7ff736000700 1 -- 172.21.15.27:0/631486386 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7ff730013e60 con 0x7ff7400fce40 2024-04-16T05:48:18.737 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:48:18.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.741+0000 7ff744e00700 1 -- 172.21.15.27:0/631486386 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff72406ec10 msgr2=0x7ff7240710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:18.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.741+0000 7ff744e00700 1 --2- 172.21.15.27:0/631486386 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff72406ec10 0x7ff7240710c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff72c004a30 tx=0x7ff72c007ab0 comp rx=0 tx=0).stop 2024-04-16T05:48:18.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.741+0000 7ff744e00700 1 -- 172.21.15.27:0/631486386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7400fce40 msgr2=0x7ff740071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:18.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.741+0000 7ff744e00700 1 --2- 172.21.15.27:0/631486386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7400fce40 0x7ff740071e40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff73000b970 tx=0x7ff730009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:18.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.741+0000 7ff744e00700 1 -- 172.21.15.27:0/631486386 shutdown_connections 2024-04-16T05:48:18.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.741+0000 7ff744e00700 1 --2- 172.21.15.27:0/631486386 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff72406ec10 0x7ff7240710c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:18.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.741+0000 7ff744e00700 1 --2- 172.21.15.27:0/631486386 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7400fce40 0x7ff740071e40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:18.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.741+0000 7ff744e00700 1 --2- 172.21.15.27:0/631486386 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7400fe040 0x7ff740070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:18.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.741+0000 7ff744e00700 1 -- 172.21.15.27:0/631486386 >> 172.21.15.27:0/631486386 conn(0x7ff7400f8530 msgr2=0x7ff7400f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:18.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.741+0000 7ff744e00700 1 -- 172.21.15.27:0/631486386 shutdown_connections 2024-04-16T05:48:18.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:18.741+0000 7ff744e00700 1 -- 172.21.15.27:0/631486386 wait complete. 2024-04-16T05:48:18.860 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:18 smithi027 ceph-mon[93635]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:48:18.860 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:18 smithi027 ceph-mon[93635]: pgmap v101: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:18.860 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:18 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1301324658' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:48:19.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:18 smithi122 ceph-mon[80930]: from='client.34284 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:48:19.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:18 smithi122 ceph-mon[80930]: pgmap v101: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:19.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:18 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1301324658' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:48:20.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:19 smithi122 ceph-mon[80930]: from='client.34296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:48:20.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:19 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/631486386' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:48:20.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:20.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:20.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:48:20.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:48:20.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:20.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:19 smithi027 ceph-mon[93635]: from='client.34296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:48:20.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:19 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/631486386' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:48:20.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:20.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:20.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:48:20.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:48:20.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:20.903 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:20 smithi027 ceph-mon[93635]: pgmap v102: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:20.903 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:20 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:48:20.904 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:20 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:48:20.904 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:20 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:48:20.904 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:20 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:48:20.904 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:20 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-16T05:48:21.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:20 smithi122 ceph-mon[80930]: pgmap v102: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:21.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:20 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:48:21.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:20 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:48:21.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:20 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:48:21.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:20 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:48:21.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:20 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-16T05:48:21.980 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:21 smithi027 ceph-mon[93635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-16T05:48:21.980 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:21 smithi027 ceph-mon[93635]: Upgrade: osd.2 is safe to restart 2024-04-16T05:48:21.980 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:21 smithi027 ceph-mon[93635]: pgmap v103: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:22.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:21 smithi122 ceph-mon[80930]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-16T05:48:22.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:21 smithi122 ceph-mon[80930]: Upgrade: osd.2 is safe to restart 2024-04-16T05:48:22.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:21 smithi122 ceph-mon[80930]: pgmap v103: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:23.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:23 smithi027 ceph-mon[93635]: Upgrade: Updating osd.2 2024-04-16T05:48:23.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:23 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:23.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:23 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T05:48:23.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:23 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:48:23.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:23 smithi027 ceph-mon[93635]: Deploying daemon osd.2 on smithi027 2024-04-16T05:48:23.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:23 smithi122 ceph-mon[80930]: Upgrade: Updating osd.2 2024-04-16T05:48:23.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:23 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:23.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:23 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-16T05:48:23.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:23 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:48:23.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:23 smithi122 ceph-mon[80930]: Deploying daemon osd.2 on smithi027 2024-04-16T05:48:24.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:24 smithi122 ceph-mon[80930]: pgmap v104: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:24.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:24.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:24 smithi027 ceph-mon[93635]: pgmap v104: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:24.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:26.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:26 smithi122 ceph-mon[80930]: pgmap v105: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:26.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:26 smithi027 ceph-mon[93635]: pgmap v105: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:27.307 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:27 smithi027 systemd[1]: Stopping Ceph osd.2 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T05:48:27.807 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:27 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2[62563]: 2024-04-16T05:48:27.540+0000 7f5e90200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T05:48:27.807 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:27 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2[62563]: 2024-04-16T05:48:27.540+0000 7f5e90200700 -1 osd.2 63 *** Got signal Terminated *** 2024-04-16T05:48:27.807 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:27 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2[62563]: 2024-04-16T05:48:27.540+0000 7f5e90200700 -1 osd.2 63 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T05:48:28.437 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:28 smithi027 ceph-mon[93635]: pgmap v106: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:28.437 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:28 smithi027 ceph-mon[93635]: osd.2 marked itself down and dead 2024-04-16T05:48:28.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:28 smithi122 ceph-mon[80930]: pgmap v106: 65 pgs: 65 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:28.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:28 smithi122 ceph-mon[80930]: osd.2 marked itself down and dead 2024-04-16T05:48:29.025 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:28 smithi027 podman[113311]: 2024-04-16 05:48:28.734000893 +0000 UTC m=+1.345714927 container died 6212f6a18fa4b7ee9200e3e1fa5852f839f57c763c82d0658dd66ad339aafeef (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2, maintainer=Guillaume Abrioux , RELEASE=wip-64502-quincy-3-45317f9, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T05:48:29.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:29 smithi027 ceph-mon[93635]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T05:48:29.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:29 smithi027 ceph-mon[93635]: osdmap e64: 6 total, 5 up, 6 in 2024-04-16T05:48:29.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:29.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:48:29.307 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:29 smithi027 podman[113311]: 2024-04-16 05:48:29.159665797 +0000 UTC m=+1.771379851 container cleanup 6212f6a18fa4b7ee9200e3e1fa5852f839f57c763c82d0658dd66ad339aafeef (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-64502-quincy-3-45317f9) 2024-04-16T05:48:29.307 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:29 smithi027 bash[113311]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2 2024-04-16T05:48:29.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:29 smithi122 ceph-mon[80930]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T05:48:29.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:29 smithi122 ceph-mon[80930]: osdmap e64: 6 total, 5 up, 6 in 2024-04-16T05:48:29.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:29.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:48:29.806 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:29 smithi027 podman[113331]: 2024-04-16 05:48:29.43555682 +0000 UTC m=+0.684213943 container remove 6212f6a18fa4b7ee9200e3e1fa5852f839f57c763c82d0658dd66ad339aafeef (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-64502-quincy-3-45317f9, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0) 2024-04-16T05:48:30.497 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:30 smithi027 ceph-mon[93635]: pgmap v108: 65 pgs: 8 peering, 8 stale+active+clean, 49 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:30.497 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:30 smithi027 ceph-mon[93635]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-16T05:48:30.497 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:30 smithi027 ceph-mon[93635]: osdmap e65: 6 total, 5 up, 6 in 2024-04-16T05:48:30.497 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:30 smithi027 podman[113420]: 2024-04-16 05:48:30.188932289 +0000 UTC m=+0.039853670 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T05:48:30.498 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:30 smithi027 podman[113420]: 2024-04-16 05:48:30.336589005 +0000 UTC m=+0.187510412 container create 7cb862284371dd2551cb902c2ca98e636267275b09d6fbf15c50e570c688c5a1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-16T05:48:30.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:30 smithi122 ceph-mon[80930]: pgmap v108: 65 pgs: 8 peering, 8 stale+active+clean, 49 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:30.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:30 smithi122 ceph-mon[80930]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-16T05:48:30.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:30 smithi122 ceph-mon[80930]: osdmap e65: 6 total, 5 up, 6 in 2024-04-16T05:48:30.807 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:30 smithi027 podman[113420]: 2024-04-16 05:48:30.609541341 +0000 UTC m=+0.460462791 container init 7cb862284371dd2551cb902c2ca98e636267275b09d6fbf15c50e570c688c5a1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, ceph=True, org.label-schema.license=GPLv2) 2024-04-16T05:48:30.807 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:30 smithi027 podman[113420]: 2024-04-16 05:48:30.622576036 +0000 UTC m=+0.473497473 container start 7cb862284371dd2551cb902c2ca98e636267275b09d6fbf15c50e570c688c5a1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240409) 2024-04-16T05:48:30.807 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:30 smithi027 podman[113420]: 2024-04-16 05:48:30.747566226 +0000 UTC m=+0.598487630 container attach 7cb862284371dd2551cb902c2ca98e636267275b09d6fbf15c50e570c688c5a1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-16T05:48:31.479 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:31 smithi027 ceph-mon[93635]: Health check failed: Degraded data redundancy: 2/72 objects degraded (2.778%), 2 pgs degraded (PG_DEGRADED) 2024-04-16T05:48:31.479 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:31 smithi027 podman[113420]: 2024-04-16 05:48:31.10280452 +0000 UTC m=+0.953725950 container died 7cb862284371dd2551cb902c2ca98e636267275b09d6fbf15c50e570c688c5a1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-16T05:48:31.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:31 smithi122 ceph-mon[80930]: Health check failed: Degraded data redundancy: 2/72 objects degraded (2.778%), 2 pgs degraded (PG_DEGRADED) 2024-04-16T05:48:32.187 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:31 smithi027 podman[113446]: 2024-04-16 05:48:31.926532068 +0000 UTC m=+0.800150177 container remove 7cb862284371dd2551cb902c2ca98e636267275b09d6fbf15c50e570c688c5a1 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-16T05:48:32.187 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:31 smithi027 systemd[1]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-16T05:48:32.187 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:31 smithi027 systemd[1]: Stopped Ceph osd.2 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968. 2024-04-16T05:48:32.188 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:31 smithi027 systemd[1]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.2.service: Consumed 14.123s CPU time. 2024-04-16T05:48:32.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:32 smithi122 ceph-mon[80930]: pgmap v110: 65 pgs: 2 active+undersized+degraded, 8 peering, 6 stale+active+clean, 9 active+undersized, 40 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2/72 objects degraded (2.778%) 2024-04-16T05:48:32.556 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:32 smithi027 systemd[1]: Starting Ceph osd.2 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T05:48:32.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:32 smithi027 ceph-mon[93635]: pgmap v110: 65 pgs: 2 active+undersized+degraded, 8 peering, 6 stale+active+clean, 9 active+undersized, 40 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 2/72 objects degraded (2.778%) 2024-04-16T05:48:33.057 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:32 smithi027 podman[113544]: 2024-04-16 05:48:32.668508984 +0000 UTC m=+0.038735422 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T05:48:33.057 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:32 smithi027 podman[113544]: 2024-04-16 05:48:32.769073542 +0000 UTC m=+0.139300011 container create 6f6a3dc009f2da0b4494a63b7d57530bf663c3f7ac72e2df2e0d7cd88a1c449f (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-activate, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T05:48:33.363 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:33 smithi027 podman[113544]: 2024-04-16 05:48:33.066333759 +0000 UTC m=+0.436560275 container init 6f6a3dc009f2da0b4494a63b7d57530bf663c3f7ac72e2df2e0d7cd88a1c449f (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240409) 2024-04-16T05:48:33.363 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:33 smithi027 podman[113544]: 2024-04-16 05:48:33.079465226 +0000 UTC m=+0.449691689 container start 6f6a3dc009f2da0b4494a63b7d57530bf663c3f7ac72e2df2e0d7cd88a1c449f (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-16T05:48:33.363 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:33 smithi027 podman[113544]: 2024-04-16 05:48:33.179910598 +0000 UTC m=+0.550137050 container attach 6f6a3dc009f2da0b4494a63b7d57530bf663c3f7ac72e2df2e0d7cd88a1c449f (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-16T05:48:33.889 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 296, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 113640 2024-04-16T05:48:33.890 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 296, success, outsize: 120 2024-04-16T05:48:34.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:34 smithi027 ceph-mon[93635]: pgmap v111: 65 pgs: 4 active+undersized+degraded, 8 peering, 19 active+undersized, 34 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 5/72 objects degraded (6.944%) 2024-04-16T05:48:34.307 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:34 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-activate[113558]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T05:48:34.307 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:34 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-activate[113558]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-16T05:48:34.308 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:34 smithi027 bash[113544]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T05:48:34.308 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:34 smithi027 bash[113544]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-16T05:48:34.308 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:34 smithi027 bash[113544]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-16T05:48:34.308 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:34 smithi027 bash[113544]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-16T05:48:34.308 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:34 smithi027 bash[113544]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-16T05:48:34.308 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:34 smithi027 bash[113544]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T05:48:34.308 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:34 smithi027 bash[113544]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-16T05:48:34.308 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:34 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-activate[113558]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-16T05:48:34.308 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:34 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-activate[113558]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-16T05:48:34.308 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:34 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-activate[113558]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-16T05:48:34.308 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:34 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-activate[113558]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T05:48:34.308 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:34 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-activate[113558]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-16T05:48:34.308 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:34 smithi027 podman[113544]: 2024-04-16 05:48:34.090278917 +0000 UTC m=+1.460505419 container died 6f6a3dc009f2da0b4494a63b7d57530bf663c3f7ac72e2df2e0d7cd88a1c449f (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-04-16T05:48:34.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:34 smithi122 ceph-mon[80930]: pgmap v111: 65 pgs: 4 active+undersized+degraded, 8 peering, 19 active+undersized, 34 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 5/72 objects degraded (6.944%) 2024-04-16T05:48:35.246 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:34 smithi027 podman[113683]: 2024-04-16 05:48:34.934561447 +0000 UTC m=+0.822307158 container remove 6f6a3dc009f2da0b4494a63b7d57530bf663c3f7ac72e2df2e0d7cd88a1c449f (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409) 2024-04-16T05:48:35.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:35 smithi122 ceph-mon[80930]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-16T05:48:35.535 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:35 smithi027 ceph-mon[93635]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-16T05:48:35.535 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:35 smithi027 podman[113744]: 2024-04-16 05:48:35.225749302 +0000 UTC m=+0.041780024 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T05:48:35.535 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:35 smithi027 podman[113744]: 2024-04-16 05:48:35.37902933 +0000 UTC m=+0.195060076 container create 3524731d727794ce24e2c35de1c0358ac3a988f5a11249b28485ff02a89e67b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T05:48:36.056 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:35 smithi027 podman[113744]: 2024-04-16 05:48:35.663920835 +0000 UTC m=+0.479951625 container init 3524731d727794ce24e2c35de1c0358ac3a988f5a11249b28485ff02a89e67b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2, ceph=True, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-16T05:48:36.057 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:35 smithi027 podman[113744]: 2024-04-16 05:48:35.674627088 +0000 UTC m=+0.490657840 container start 3524731d727794ce24e2c35de1c0358ac3a988f5a11249b28485ff02a89e67b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T05:48:36.057 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:35 smithi027 ceph-osd[113762]: -- 172.21.15.27:0/185210574 <== mon.0 v2:172.21.15.27:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5573a9599680 con 0x5573a9562c00 2024-04-16T05:48:36.057 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:35 smithi027 bash[113744]: 3524731d727794ce24e2c35de1c0358ac3a988f5a11249b28485ff02a89e67b7 2024-04-16T05:48:36.057 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:35 smithi027 systemd[1]: Started Ceph osd.2 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968. 2024-04-16T05:48:36.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:36 smithi122 ceph-mon[80930]: pgmap v112: 65 pgs: 7 active+undersized+degraded, 24 active+undersized, 34 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 12/72 objects degraded (16.667%) 2024-04-16T05:48:36.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:36 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:36.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:36 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:36.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:36 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:48:36.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:36 smithi027 ceph-mon[93635]: pgmap v112: 65 pgs: 7 active+undersized+degraded, 24 active+undersized, 34 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 12/72 objects degraded (16.667%) 2024-04-16T05:48:36.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:36 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:36.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:36 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:36.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:36 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:48:36.557 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:36 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2[113758]: 2024-04-16T05:48:36.255+0000 7f843b1ed7c0 -1 Falling back to public interface 2024-04-16T05:48:38.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:38 smithi122 ceph-mon[80930]: pgmap v113: 65 pgs: 7 active+undersized+degraded, 24 active+undersized, 34 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 12/72 objects degraded (16.667%) 2024-04-16T05:48:38.524 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:38 smithi027 ceph-mon[93635]: pgmap v113: 65 pgs: 7 active+undersized+degraded, 24 active+undersized, 34 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 12/72 objects degraded (16.667%) 2024-04-16T05:48:39.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:39 smithi122 ceph-mon[80930]: Health check update: Degraded data redundancy: 12/72 objects degraded (16.667%), 7 pgs degraded (PG_DEGRADED) 2024-04-16T05:48:39.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:39 smithi027 ceph-mon[93635]: Health check update: Degraded data redundancy: 12/72 objects degraded (16.667%), 7 pgs degraded (PG_DEGRADED) 2024-04-16T05:48:40.273 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:39 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2[113758]: 2024-04-16T05:48:39.932+0000 7f843b1ed7c0 -1 osd.2 63 log_to_monitors true 2024-04-16T05:48:40.556 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 05:48:40 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2[113758]: 2024-04-16T05:48:40.301+0000 7f842a600700 -1 osd.2 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T05:48:40.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:40 smithi027 ceph-mon[93635]: pgmap v114: 65 pgs: 7 active+undersized+degraded, 24 active+undersized, 34 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 12/72 objects degraded (16.667%) 2024-04-16T05:48:40.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:40 smithi027 ceph-mon[93635]: from='osd.2 [v2:172.21.15.27:6818/1424147666,v1:172.21.15.27:6819/1424147666]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-16T05:48:40.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:40 smithi122 ceph-mon[80930]: pgmap v114: 65 pgs: 7 active+undersized+degraded, 24 active+undersized, 34 active+clean; 584 KiB data, 1.2 GiB used, 535 GiB / 536 GiB avail; 12/72 objects degraded (16.667%) 2024-04-16T05:48:40.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:40 smithi122 ceph-mon[80930]: from='osd.2 [v2:172.21.15.27:6818/1424147666,v1:172.21.15.27:6819/1424147666]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-16T05:48:41.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:41 smithi027 ceph-mon[93635]: from='osd.2 [v2:172.21.15.27:6818/1424147666,v1:172.21.15.27:6819/1424147666]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-16T05:48:41.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:41 smithi027 ceph-mon[93635]: osdmap e66: 6 total, 5 up, 6 in 2024-04-16T05:48:41.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:41 smithi027 ceph-mon[93635]: from='osd.2 [v2:172.21.15.27:6818/1424147666,v1:172.21.15.27:6819/1424147666]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-16T05:48:41.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:41 smithi122 ceph-mon[80930]: from='osd.2 [v2:172.21.15.27:6818/1424147666,v1:172.21.15.27:6819/1424147666]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-16T05:48:41.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:41 smithi122 ceph-mon[80930]: osdmap e66: 6 total, 5 up, 6 in 2024-04-16T05:48:41.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:41 smithi122 ceph-mon[80930]: from='osd.2 [v2:172.21.15.27:6818/1424147666,v1:172.21.15.27:6819/1424147666]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi027", "root=default"]}]: dispatch 2024-04-16T05:48:42.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:42 smithi027 ceph-mon[93635]: pgmap v116: 65 pgs: 7 active+undersized+degraded, 24 active+undersized, 34 active+clean; 584 KiB data, 957 MiB used, 535 GiB / 536 GiB avail; 12/72 objects degraded (16.667%) 2024-04-16T05:48:42.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:42 smithi027 ceph-mon[93635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T05:48:42.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:42 smithi027 ceph-mon[93635]: osd.2 [v2:172.21.15.27:6818/1424147666,v1:172.21.15.27:6819/1424147666] boot 2024-04-16T05:48:42.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:42 smithi027 ceph-mon[93635]: osdmap e67: 6 total, 6 up, 6 in 2024-04-16T05:48:42.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:42 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T05:48:42.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:42 smithi122 ceph-mon[80930]: pgmap v116: 65 pgs: 7 active+undersized+degraded, 24 active+undersized, 34 active+clean; 584 KiB data, 957 MiB used, 535 GiB / 536 GiB avail; 12/72 objects degraded (16.667%) 2024-04-16T05:48:42.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:42 smithi122 ceph-mon[80930]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T05:48:42.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:42 smithi122 ceph-mon[80930]: osd.2 [v2:172.21.15.27:6818/1424147666,v1:172.21.15.27:6819/1424147666] boot 2024-04-16T05:48:42.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:42 smithi122 ceph-mon[80930]: osdmap e67: 6 total, 6 up, 6 in 2024-04-16T05:48:42.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:42 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-16T05:48:43.505 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:43 smithi027 ceph-mon[93635]: osdmap e68: 6 total, 6 up, 6 in 2024-04-16T05:48:43.505 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:43 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:43.506 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:43 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:43.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:43 smithi122 ceph-mon[80930]: osdmap e68: 6 total, 6 up, 6 in 2024-04-16T05:48:43.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:43 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:43.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:43 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:44.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:44 smithi027 ceph-mon[93635]: pgmap v119: 65 pgs: 6 peering, 6 active+undersized+degraded, 19 active+undersized, 34 active+clean; 584 KiB data, 957 MiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T05:48:44.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:44.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:48:44.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:44 smithi027 ceph-mon[93635]: Health check update: Degraded data redundancy: 10/72 objects degraded (13.889%), 6 pgs degraded (PG_DEGRADED) 2024-04-16T05:48:44.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:44 smithi122 ceph-mon[80930]: pgmap v119: 65 pgs: 6 peering, 6 active+undersized+degraded, 19 active+undersized, 34 active+clean; 584 KiB data, 957 MiB used, 535 GiB / 536 GiB avail; 10/72 objects degraded (13.889%) 2024-04-16T05:48:44.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:44 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:44.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:44 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:48:44.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:44 smithi122 ceph-mon[80930]: Health check update: Degraded data redundancy: 10/72 objects degraded (13.889%), 6 pgs degraded (PG_DEGRADED) 2024-04-16T05:48:46.704 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:46 smithi027 ceph-mon[93635]: pgmap v120: 65 pgs: 6 peering, 2 active+undersized+degraded, 10 active+undersized, 47 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail; 5/72 objects degraded (6.944%) 2024-04-16T05:48:46.704 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:46 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:46.704 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:46 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:46.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:46 smithi122 ceph-mon[80930]: pgmap v120: 65 pgs: 6 peering, 2 active+undersized+degraded, 10 active+undersized, 47 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail; 5/72 objects degraded (6.944%) 2024-04-16T05:48:46.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:46 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:46.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:46 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:47.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:47 smithi122 ceph-mon[80930]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/72 objects degraded (6.944%), 2 pgs degraded) 2024-04-16T05:48:47.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:47 smithi122 ceph-mon[80930]: Cluster is now healthy 2024-04-16T05:48:47.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:47 smithi027 ceph-mon[93635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/72 objects degraded (6.944%), 2 pgs degraded) 2024-04-16T05:48:47.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:47 smithi027 ceph-mon[93635]: Cluster is now healthy 2024-04-16T05:48:48.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:48 smithi122 ceph-mon[80930]: pgmap v121: 65 pgs: 65 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:48.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:48 smithi027 ceph-mon[93635]: pgmap v121: 65 pgs: 65 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:48.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.912+0000 7f1e34400700 1 -- 172.21.15.27:0/3170400598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e2c070870 msgr2=0x7f1e2c070ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:48.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.912+0000 7f1e34400700 1 --2- 172.21.15.27:0/3170400598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e2c070870 0x7f1e2c070ce0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1e18004940 tx=0x7f1e18009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:48.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.912+0000 7f1e34400700 1 -- 172.21.15.27:0/3170400598 shutdown_connections 2024-04-16T05:48:48.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.912+0000 7f1e34400700 1 --2- 172.21.15.27:0/3170400598 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e2c070870 0x7f1e2c070ce0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:48.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.912+0000 7f1e34400700 1 --2- 172.21.15.27:0/3170400598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e2c071e20 0x7f1e2c070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:48.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.912+0000 7f1e34400700 1 -- 172.21.15.27:0/3170400598 >> 172.21.15.27:0/3170400598 conn(0x7f1e2c0f8580 msgr2=0x7f1e2c0fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.912+0000 7f1e34400700 1 -- 172.21.15.27:0/3170400598 shutdown_connections 2024-04-16T05:48:48.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.912+0000 7f1e34400700 1 -- 172.21.15.27:0/3170400598 wait complete. 2024-04-16T05:48:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.913+0000 7f1e34400700 1 Processor -- start 2024-04-16T05:48:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.913+0000 7f1e34400700 1 -- start start 2024-04-16T05:48:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.913+0000 7f1e34400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e2c070870 0x7f1e2c10fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.913+0000 7f1e34400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e2c071e20 0x7f1e2c1100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.913+0000 7f1e34400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c1106a0 con 0x7f1e2c070870 2024-04-16T05:48:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.913+0000 7f1e34400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c110810 con 0x7f1e2c071e20 2024-04-16T05:48:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.914+0000 7f1e31a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e2c070870 0x7f1e2c10fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.914+0000 7f1e31a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e2c070870 0x7f1e2c10fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44576/0 (socket says 172.21.15.27:44576) 2024-04-16T05:48:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.914+0000 7f1e31a00700 1 -- 172.21.15.27:0/3917464090 learned_addr learned my addr 172.21.15.27:0/3917464090 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:48:48.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.914+0000 7f1e31000700 1 --2- 172.21.15.27:0/3917464090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e2c071e20 0x7f1e2c1100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.914+0000 7f1e31a00700 1 -- 172.21.15.27:0/3917464090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e2c071e20 msgr2=0x7f1e2c1100a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.914+0000 7f1e31a00700 1 --2- 172.21.15.27:0/3917464090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e2c071e20 0x7f1e2c1100a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.914+0000 7f1e31a00700 1 -- 172.21.15.27:0/3917464090 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e18005040 con 0x7f1e2c070870 2024-04-16T05:48:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.915+0000 7f1e31a00700 1 --2- 172.21.15.27:0/3917464090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e2c070870 0x7f1e2c10fb60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1e20002a50 tx=0x7f1e20004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.915+0000 7f1e29600700 1 -- 172.21.15.27:0/3917464090 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e20007cc0 con 0x7f1e2c070870 2024-04-16T05:48:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.915+0000 7f1e34400700 1 -- 172.21.15.27:0/3917464090 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2c115270 con 0x7f1e2c070870 2024-04-16T05:48:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.915+0000 7f1e29600700 1 -- 172.21.15.27:0/3917464090 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e2000f040 con 0x7f1e2c070870 2024-04-16T05:48:48.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.915+0000 7f1e29600700 1 -- 172.21.15.27:0/3917464090 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e2000da40 con 0x7f1e2c070870 2024-04-16T05:48:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.915+0000 7f1e34400700 1 -- 172.21.15.27:0/3917464090 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e2c115830 con 0x7f1e2c070870 2024-04-16T05:48:48.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.917+0000 7f1e34400700 1 -- 172.21.15.27:0/3917464090 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e2c04c710 con 0x7f1e2c070870 2024-04-16T05:48:48.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.918+0000 7f1e29600700 1 -- 172.21.15.27:0/3917464090 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1e200097c0 con 0x7f1e2c070870 2024-04-16T05:48:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.919+0000 7f1e29600700 1 --2- 172.21.15.27:0/3917464090 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1e1406ec00 0x7f1e140710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.919+0000 7f1e29600700 1 -- 172.21.15.27:0/3917464090 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f1e2008e760 con 0x7f1e2c070870 2024-04-16T05:48:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.919+0000 7f1e31000700 1 --2- 172.21.15.27:0/3917464090 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1e1406ec00 0x7f1e140710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:48.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.920+0000 7f1e31000700 1 --2- 172.21.15.27:0/3917464090 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1e1406ec00 0x7f1e140710b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1e18004f90 tx=0x7f1e18009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:48.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:48.924+0000 7f1e29600700 1 -- 172.21.15.27:0/3917464090 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1e2005ac40 con 0x7f1e2c070870 2024-04-16T05:48:49.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.182+0000 7f1e34400700 1 -- 172.21.15.27:0/3917464090 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e2c103e70 con 0x7f1e1406ec00 2024-04-16T05:48:49.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.185+0000 7f1e29600700 1 -- 172.21.15.27:0/3917464090 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f1e2c103e70 con 0x7f1e1406ec00 2024-04-16T05:48:49.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.188+0000 7f1e34400700 1 -- 172.21.15.27:0/3917464090 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1e1406ec00 msgr2=0x7f1e140710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:49.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.188+0000 7f1e34400700 1 --2- 172.21.15.27:0/3917464090 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1e1406ec00 0x7f1e140710b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1e18004f90 tx=0x7f1e18009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:49.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.188+0000 7f1e34400700 1 -- 172.21.15.27:0/3917464090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e2c070870 msgr2=0x7f1e2c10fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:49.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.188+0000 7f1e34400700 1 --2- 172.21.15.27:0/3917464090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e2c070870 0x7f1e2c10fb60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1e20002a50 tx=0x7f1e20004600 comp rx=0 tx=0).stop 2024-04-16T05:48:49.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.188+0000 7f1e34400700 1 -- 172.21.15.27:0/3917464090 shutdown_connections 2024-04-16T05:48:49.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.188+0000 7f1e34400700 1 --2- 172.21.15.27:0/3917464090 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1e1406ec00 0x7f1e140710b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:49.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.188+0000 7f1e34400700 1 --2- 172.21.15.27:0/3917464090 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e2c070870 0x7f1e2c10fb60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:49.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.188+0000 7f1e34400700 1 --2- 172.21.15.27:0/3917464090 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e2c071e20 0x7f1e2c1100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:49.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.188+0000 7f1e34400700 1 -- 172.21.15.27:0/3917464090 >> 172.21.15.27:0/3917464090 conn(0x7f1e2c0f8580 msgr2=0x7f1e2c0f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:49.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.189+0000 7f1e34400700 1 -- 172.21.15.27:0/3917464090 shutdown_connections 2024-04-16T05:48:49.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.189+0000 7f1e34400700 1 -- 172.21.15.27:0/3917464090 wait complete. 2024-04-16T05:48:49.206 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:48:49.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.359+0000 7f9256e00700 1 -- 172.21.15.27:0/1244219272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92500fdd60 msgr2=0x7f92501001e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:49.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.359+0000 7f9256e00700 1 --2- 172.21.15.27:0/1244219272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92500fdd60 0x7f92501001e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f923c004940 tx=0x7f923c009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:49.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.359+0000 7f9256e00700 1 -- 172.21.15.27:0/1244219272 shutdown_connections 2024-04-16T05:48:49.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.359+0000 7f9256e00700 1 --2- 172.21.15.27:0/1244219272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92500fdd60 0x7f92501001e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:49.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.359+0000 7f9256e00700 1 --2- 172.21.15.27:0/1244219272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92500662f0 0x7f92500fd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:49.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.359+0000 7f9256e00700 1 -- 172.21.15.27:0/1244219272 >> 172.21.15.27:0/1244219272 conn(0x7f92500f74c0 msgr2=0x7f92500f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:49.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.359+0000 7f9256e00700 1 -- 172.21.15.27:0/1244219272 shutdown_connections 2024-04-16T05:48:49.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.360+0000 7f9256e00700 1 -- 172.21.15.27:0/1244219272 wait complete. 2024-04-16T05:48:49.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.360+0000 7f9256e00700 1 Processor -- start 2024-04-16T05:48:49.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.361+0000 7f9256e00700 1 -- start start 2024-04-16T05:48:49.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.361+0000 7f9256e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92500662f0 0x7f9250114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:49.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.361+0000 7f9256e00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92500fdd60 0x7f92501145a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:49.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.361+0000 7f9256e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9250114ba0 con 0x7f92500662f0 2024-04-16T05:48:49.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.361+0000 7f9256e00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92501153f0 con 0x7f92500fdd60 2024-04-16T05:48:49.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.361+0000 7f924fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92500662f0 0x7f9250114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:49.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.361+0000 7f924fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92500662f0 0x7f9250114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44588/0 (socket says 172.21.15.27:44588) 2024-04-16T05:48:49.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.361+0000 7f924fe00700 1 -- 172.21.15.27:0/3364569686 learned_addr learned my addr 172.21.15.27:0/3364569686 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:48:49.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.362+0000 7f924f400700 1 --2- 172.21.15.27:0/3364569686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92500fdd60 0x7f92501145a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:49.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.362+0000 7f924fe00700 1 -- 172.21.15.27:0/3364569686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92500fdd60 msgr2=0x7f92501145a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:49.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.362+0000 7f924fe00700 1 --2- 172.21.15.27:0/3364569686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92500fdd60 0x7f92501145a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:49.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.362+0000 7f924fe00700 1 -- 172.21.15.27:0/3364569686 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f923c005040 con 0x7f92500662f0 2024-04-16T05:48:49.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.362+0000 7f924fe00700 1 --2- 172.21.15.27:0/3364569686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92500662f0 0x7f9250114060 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9240002a50 tx=0x7f9240004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:49.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.362+0000 7f9247e00700 1 -- 172.21.15.27:0/3364569686 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9240007cc0 con 0x7f92500662f0 2024-04-16T05:48:49.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.363+0000 7f9247e00700 1 -- 172.21.15.27:0/3364569686 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f924000f040 con 0x7f92500662f0 2024-04-16T05:48:49.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.363+0000 7f9247e00700 1 -- 172.21.15.27:0/3364569686 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f924000daf0 con 0x7f92500662f0 2024-04-16T05:48:49.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.363+0000 7f9256e00700 1 -- 172.21.15.27:0/3364569686 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92501156d0 con 0x7f92500662f0 2024-04-16T05:48:49.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.363+0000 7f9256e00700 1 -- 172.21.15.27:0/3364569686 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9250115bc0 con 0x7f92500662f0 2024-04-16T05:48:49.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.364+0000 7f9256e00700 1 -- 172.21.15.27:0/3364569686 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f925005f5f0 con 0x7f92500662f0 2024-04-16T05:48:49.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.365+0000 7f9247e00700 1 -- 172.21.15.27:0/3364569686 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9240009810 con 0x7f92500662f0 2024-04-16T05:48:49.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.366+0000 7f9247e00700 1 --2- 172.21.15.27:0/3364569686 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f923406ec10 0x7f92340710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:49.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.366+0000 7f9247e00700 1 -- 172.21.15.27:0/3364569686 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f924008e790 con 0x7f92500662f0 2024-04-16T05:48:49.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.367+0000 7f924f400700 1 --2- 172.21.15.27:0/3364569686 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f923406ec10 0x7f92340710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:49.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.367+0000 7f924f400700 1 --2- 172.21.15.27:0/3364569686 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f923406ec10 0x7f92340710c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f923c004f90 tx=0x7f923c009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:49.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.375+0000 7f9247e00700 1 -- 172.21.15.27:0/3364569686 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f924005af00 con 0x7f92500662f0 2024-04-16T05:48:49.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.634+0000 7f9256e00700 1 -- 172.21.15.27:0/3364569686 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9250115fb0 con 0x7f923406ec10 2024-04-16T05:48:49.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.635+0000 7f9247e00700 1 -- 172.21.15.27:0/3364569686 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f9250115fb0 con 0x7f923406ec10 2024-04-16T05:48:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.638+0000 7f9256e00700 1 -- 172.21.15.27:0/3364569686 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f923406ec10 msgr2=0x7f92340710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.638+0000 7f9256e00700 1 --2- 172.21.15.27:0/3364569686 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f923406ec10 0x7f92340710c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f923c004f90 tx=0x7f923c009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.639+0000 7f9256e00700 1 -- 172.21.15.27:0/3364569686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92500662f0 msgr2=0x7f9250114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.639+0000 7f9256e00700 1 --2- 172.21.15.27:0/3364569686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92500662f0 0x7f9250114060 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9240002a50 tx=0x7f9240004600 comp rx=0 tx=0).stop 2024-04-16T05:48:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.639+0000 7f9256e00700 1 -- 172.21.15.27:0/3364569686 shutdown_connections 2024-04-16T05:48:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.639+0000 7f9256e00700 1 --2- 172.21.15.27:0/3364569686 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f923406ec10 0x7f92340710c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.639+0000 7f9256e00700 1 --2- 172.21.15.27:0/3364569686 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92500662f0 0x7f9250114060 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.639+0000 7f9256e00700 1 --2- 172.21.15.27:0/3364569686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f92500fdd60 0x7f92501145a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.639+0000 7f9256e00700 1 -- 172.21.15.27:0/3364569686 >> 172.21.15.27:0/3364569686 conn(0x7f92500f74c0 msgr2=0x7f92500f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.639+0000 7f9256e00700 1 -- 172.21.15.27:0/3364569686 shutdown_connections 2024-04-16T05:48:49.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.639+0000 7f9256e00700 1 -- 172.21.15.27:0/3364569686 wait complete. 2024-04-16T05:48:49.647 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 298, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115760 2024-04-16T05:48:49.648 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 298, success, outsize: 120 2024-04-16T05:48:49.665 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 300, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115853 2024-04-16T05:48:49.665 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 300, success, outsize: 120 2024-04-16T05:48:49.691 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 302, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115948 2024-04-16T05:48:49.691 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 302, success, outsize: 120 2024-04-16T05:48:49.710 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 304, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116049 2024-04-16T05:48:49.710 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 304, success, outsize: 120 2024-04-16T05:48:49.730 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 306, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116147 2024-04-16T05:48:49.730 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 306, success, outsize: 120 2024-04-16T05:48:49.750 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 308, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116239 2024-04-16T05:48:49.750 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 308, success, outsize: 120 2024-04-16T05:48:49.772 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 310, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116334 2024-04-16T05:48:49.772 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 310, success, outsize: 120 2024-04-16T05:48:49.793 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 312, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116436 2024-04-16T05:48:49.793 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 312, success, outsize: 120 2024-04-16T05:48:49.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.810+0000 7f78cb400700 1 -- 172.21.15.27:0/1002600012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c4071e50 msgr2=0x7f78c4070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:49.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.810+0000 7f78cb400700 1 --2- 172.21.15.27:0/1002600012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c4071e50 0x7f78c4070350 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f78b8004940 tx=0x7f78b8009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:49.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.810+0000 7f78cb400700 1 -- 172.21.15.27:0/1002600012 shutdown_connections 2024-04-16T05:48:49.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.810+0000 7f78cb400700 1 --2- 172.21.15.27:0/1002600012 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78c4070890 0x7f78c4070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:49.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.810+0000 7f78cb400700 1 --2- 172.21.15.27:0/1002600012 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c4071e50 0x7f78c4070350 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:49.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.810+0000 7f78cb400700 1 -- 172.21.15.27:0/1002600012 >> 172.21.15.27:0/1002600012 conn(0x7f78c40f8540 msgr2=0x7f78c40fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:49.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.810+0000 7f78cb400700 1 -- 172.21.15.27:0/1002600012 shutdown_connections 2024-04-16T05:48:49.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.810+0000 7f78cb400700 1 -- 172.21.15.27:0/1002600012 wait complete. 2024-04-16T05:48:49.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.811+0000 7f78cb400700 1 Processor -- start 2024-04-16T05:48:49.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.811+0000 7f78cb400700 1 -- start start 2024-04-16T05:48:49.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.812+0000 7f78cb400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c4070890 0x7f78c410fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:49.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.812+0000 7f78cb400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78c4071e50 0x7f78c4110260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:49.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.812+0000 7f78cb400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c4110860 con 0x7f78c4070890 2024-04-16T05:48:49.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.812+0000 7f78cb400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c4110fa0 con 0x7f78c4071e50 2024-04-16T05:48:49.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.812+0000 7f78c8a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c4070890 0x7f78c410fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:49.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.812+0000 7f78c8a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c4070890 0x7f78c410fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44614/0 (socket says 172.21.15.27:44614) 2024-04-16T05:48:49.812 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.812+0000 7f78c8a00700 1 -- 172.21.15.27:0/3840443024 learned_addr learned my addr 172.21.15.27:0/3840443024 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:48:49.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.812+0000 7f78c3e00700 1 --2- 172.21.15.27:0/3840443024 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78c4071e50 0x7f78c4110260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:49.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.813+0000 7f78c8a00700 1 -- 172.21.15.27:0/3840443024 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78c4071e50 msgr2=0x7f78c4110260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:49.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.813+0000 7f78c8a00700 1 --2- 172.21.15.27:0/3840443024 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78c4071e50 0x7f78c4110260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:49.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.813+0000 7f78c8a00700 1 -- 172.21.15.27:0/3840443024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b8005040 con 0x7f78c4070890 2024-04-16T05:48:49.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.813+0000 7f78c3e00700 1 --2- 172.21.15.27:0/3840443024 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78c4071e50 0x7f78c4110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T05:48:49.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.813+0000 7f78c8a00700 1 --2- 172.21.15.27:0/3840443024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c4070890 0x7f78c410fd20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f78b800b970 tx=0x7f78b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:49.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.814+0000 7f78c0c00700 1 -- 172.21.15.27:0/3840443024 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78b800e040 con 0x7f78c4070890 2024-04-16T05:48:49.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.814+0000 7f78c0c00700 1 -- 172.21.15.27:0/3840443024 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f78b800bbf0 con 0x7f78c4070890 2024-04-16T05:48:49.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.814+0000 7f78cb400700 1 -- 172.21.15.27:0/3840443024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78c4111220 con 0x7f78c4070890 2024-04-16T05:48:49.813 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.814+0000 7f78c0c00700 1 -- 172.21.15.27:0/3840443024 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f78b801c630 con 0x7f78c4070890 2024-04-16T05:48:49.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.814+0000 7f78cb400700 1 -- 172.21.15.27:0/3840443024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c41117e0 con 0x7f78c4070890 2024-04-16T05:48:49.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.815+0000 7f78cb400700 1 -- 172.21.15.27:0/3840443024 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78c405f5f0 con 0x7f78c4070890 2024-04-16T05:48:49.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.817+0000 7f78c0c00700 1 -- 172.21.15.27:0/3840443024 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f78b800bd60 con 0x7f78c4070890 2024-04-16T05:48:49.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.818+0000 7f78c0c00700 1 --2- 172.21.15.27:0/3840443024 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f78ac06ec10 0x7f78ac0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:49.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.818+0000 7f78c0c00700 1 -- 172.21.15.27:0/3840443024 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f78b808fda0 con 0x7f78c4070890 2024-04-16T05:48:49.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.818+0000 7f78c3e00700 1 --2- 172.21.15.27:0/3840443024 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f78ac06ec10 0x7f78ac0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:49.818 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 314, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116553 2024-04-16T05:48:49.818 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 314, success, outsize: 120 2024-04-16T05:48:49.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.819+0000 7f78c3e00700 1 --2- 172.21.15.27:0/3840443024 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f78ac06ec10 0x7f78ac0710c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f78b0009fd0 tx=0x7f78b0009ee0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:49.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:49.823+0000 7f78c0c00700 1 -- 172.21.15.27:0/3840443024 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f78b805c590 con 0x7f78c4070890 2024-04-16T05:48:49.834 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 316, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116645 2024-04-16T05:48:49.834 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 316, success, outsize: 120 2024-04-16T05:48:49.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 318, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116740 2024-04-16T05:48:49.850 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 318, success, outsize: 120 2024-04-16T05:48:49.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 320, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116841 2024-04-16T05:48:49.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 320, success, outsize: 120 2024-04-16T05:48:49.885 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 322, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 116939 2024-04-16T05:48:49.885 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 322, success, outsize: 120 2024-04-16T05:48:49.902 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 324, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 117031 2024-04-16T05:48:49.902 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 324, success, outsize: 120 2024-04-16T05:48:49.920 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 326, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 117126 2024-04-16T05:48:49.920 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 326, success, outsize: 120 2024-04-16T05:48:49.940 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 328, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 117227 2024-04-16T05:48:49.940 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 328, success, outsize: 120 2024-04-16T05:48:49.959 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 330, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 117325 2024-04-16T05:48:49.959 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 330, success, outsize: 120 2024-04-16T05:48:49.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 332, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 117417 2024-04-16T05:48:49.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 332, success, outsize: 120 2024-04-16T05:48:49.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 334, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 117512 2024-04-16T05:48:49.997 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 334, success, outsize: 120 2024-04-16T05:48:50.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 336, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 117613 2024-04-16T05:48:50.018 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 336, success, outsize: 120 2024-04-16T05:48:50.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 338, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 117711 2024-04-16T05:48:50.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 338, success, outsize: 120 2024-04-16T05:48:50.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 340, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 117803 2024-04-16T05:48:50.060 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 340, success, outsize: 120 2024-04-16T05:48:50.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.079+0000 7f78cb400700 1 -- 172.21.15.27:0/3840443024 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f78c4111bd0 con 0x7f78ac06ec10 2024-04-16T05:48:50.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 342, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 117899 2024-04-16T05:48:50.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 342, success, outsize: 120 2024-04-16T05:48:50.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.091+0000 7f78c0c00700 1 -- 172.21.15.27:0/3840443024 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f78c4111bd0 con 0x7f78ac06ec10 2024-04-16T05:48:50.090 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:48:50.090 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (5m) 6s ago 15m 13.0M - 0.25.0 c8568f914cd2 4ed63a8a6b33 2024-04-16T05:48:50.090 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (117s) 6s ago 16m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 4aa1f0006685 2024-04-16T05:48:50.090 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (109s) 104s ago 14m 6656k - 19.0.0-2884-g218a6914 a4e4811815e4 1c1912507d56 2024-04-16T05:48:50.090 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (5m) 6s ago 15m 75.4M - 9.4.7 954c08fa6188 f7ee2a1afd12 2024-04-16T05:48:50.091 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (8m) 6s ago 8m 23.7M - 17.2.7-847-g45317f95 b333581ccdf5 0824e2621be5 2024-04-16T05:48:50.091 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (8m) 6s ago 8m 14.0M - 17.2.7-847-g45317f95 b333581ccdf5 b332a72cc594 2024-04-16T05:48:50.091 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (8m) 104s ago 8m 13.6M - 17.2.7-847-g45317f95 b333581ccdf5 f5e36815ddfa 2024-04-16T05:48:50.091 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (8m) 104s ago 8m 13.5M - 17.2.7-847-g45317f95 b333581ccdf5 755a73c65a7f 2024-04-16T05:48:50.091 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:8443,9283,8765 running (4m) 6s ago 17m 502M - 19.0.0-2884-g218a6914 a4e4811815e4 b69a0b08e12b 2024-04-16T05:48:50.091 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283,8765 running (3m) 104s ago 14m 451M - 19.0.0-2884-g218a6914 a4e4811815e4 32e27bcd7e42 2024-04-16T05:48:50.091 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (3m) 6s ago 17m 45.4M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 299ca337e4ca 2024-04-16T05:48:50.091 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (2m) 104s ago 14m 31.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5db5f3037ed0 2024-04-16T05:48:50.091 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (5m) 6s ago 16m 8086k - 1.5.0 0da6a335fe13 5e29e29868b0 2024-04-16T05:48:50.091 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (4m) 104s ago 14m 8032k - 1.5.0 0da6a335fe13 a360922bc621 2024-04-16T05:48:50.091 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (78s) 6s ago 13m 42.5M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 cef03b67bc5a 2024-04-16T05:48:50.091 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (46s) 6s ago 12m 42.2M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3a6ab1fc6e79 2024-04-16T05:48:50.092 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (14s) 6s ago 12m 11.1M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3524731d7277 2024-04-16T05:48:50.092 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (11m) 104s ago 11m 46.3M 2581M 17.2.7-847-g45317f95 b333581ccdf5 d6724ad05f9f 2024-04-16T05:48:50.092 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (10m) 104s ago 10m 45.0M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7a59908a3e10 2024-04-16T05:48:50.092 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (10m) 104s ago 10m 44.2M 2581M 17.2.7-847-g45317f95 b333581ccdf5 8e4b60163907 2024-04-16T05:48:50.092 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (3m) 6s ago 15m 57.0M - 2.43.0 a07b618ecd1d e70f55ba5ec0 2024-04-16T05:48:50.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.094+0000 7f78cb400700 1 -- 172.21.15.27:0/3840443024 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f78ac06ec10 msgr2=0x7f78ac0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:50.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.094+0000 7f78cb400700 1 --2- 172.21.15.27:0/3840443024 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f78ac06ec10 0x7f78ac0710c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f78b0009fd0 tx=0x7f78b0009ee0 comp rx=0 tx=0).stop 2024-04-16T05:48:50.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.094+0000 7f78cb400700 1 -- 172.21.15.27:0/3840443024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c4070890 msgr2=0x7f78c410fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:50.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.094+0000 7f78cb400700 1 --2- 172.21.15.27:0/3840443024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c4070890 0x7f78c410fd20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f78b800b970 tx=0x7f78b8009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:50.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.094+0000 7f78cb400700 1 -- 172.21.15.27:0/3840443024 shutdown_connections 2024-04-16T05:48:50.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.094+0000 7f78cb400700 1 --2- 172.21.15.27:0/3840443024 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f78ac06ec10 0x7f78ac0710c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:50.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.094+0000 7f78cb400700 1 --2- 172.21.15.27:0/3840443024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f78c4070890 0x7f78c410fd20 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:50.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.094+0000 7f78cb400700 1 --2- 172.21.15.27:0/3840443024 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f78c4071e50 0x7f78c4110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:50.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.094+0000 7f78cb400700 1 -- 172.21.15.27:0/3840443024 >> 172.21.15.27:0/3840443024 conn(0x7f78c40f8540 msgr2=0x7f78c40f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:50.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.094+0000 7f78cb400700 1 -- 172.21.15.27:0/3840443024 shutdown_connections 2024-04-16T05:48:50.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.094+0000 7f78cb400700 1 -- 172.21.15.27:0/3840443024 wait complete. 2024-04-16T05:48:50.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 344, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 118001 2024-04-16T05:48:50.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 344, success, outsize: 120 2024-04-16T05:48:50.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 346, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 118100 2024-04-16T05:48:50.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 346, success, outsize: 120 2024-04-16T05:48:50.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 348, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 118192 2024-04-16T05:48:50.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 348, success, outsize: 120 2024-04-16T05:48:50.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 350, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 118287 2024-04-16T05:48:50.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 350, success, outsize: 120 2024-04-16T05:48:50.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 352, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 118388 2024-04-16T05:48:50.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 352, success, outsize: 120 2024-04-16T05:48:50.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.250+0000 7fd816400700 1 -- 172.21.15.27:0/3797786271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd810071e40 msgr2=0x7fd810070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:50.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.250+0000 7fd816400700 1 --2- 172.21.15.27:0/3797786271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd810071e40 0x7fd810070350 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd800004940 tx=0x7fd800009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:50.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.250+0000 7fd816400700 1 -- 172.21.15.27:0/3797786271 shutdown_connections 2024-04-16T05:48:50.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.250+0000 7fd816400700 1 --2- 172.21.15.27:0/3797786271 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd810070890 0x7fd810070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:50.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.250+0000 7fd816400700 1 --2- 172.21.15.27:0/3797786271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd810071e40 0x7fd810070350 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:50.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.250+0000 7fd816400700 1 -- 172.21.15.27:0/3797786271 >> 172.21.15.27:0/3797786271 conn(0x7fd8100f8590 msgr2=0x7fd8100fa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:50.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.250+0000 7fd816400700 1 -- 172.21.15.27:0/3797786271 shutdown_connections 2024-04-16T05:48:50.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.251+0000 7fd816400700 1 -- 172.21.15.27:0/3797786271 wait complete. 2024-04-16T05:48:50.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.252+0000 7fd816400700 1 Processor -- start 2024-04-16T05:48:50.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.252+0000 7fd816400700 1 -- start start 2024-04-16T05:48:50.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.252+0000 7fd816400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd810070890 0x7fd81010fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:50.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.252+0000 7fd816400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd810071e40 0x7fd810110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:50.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.252+0000 7fd816400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd810110850 con 0x7fd810071e40 2024-04-16T05:48:50.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.252+0000 7fd816400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd810110f90 con 0x7fd810070890 2024-04-16T05:48:50.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.253+0000 7fd80ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd810071e40 0x7fd810110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:50.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.253+0000 7fd80ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd810071e40 0x7fd810110250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44640/0 (socket says 172.21.15.27:44640) 2024-04-16T05:48:50.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.253+0000 7fd80ea00700 1 -- 172.21.15.27:0/1417366722 learned_addr learned my addr 172.21.15.27:0/1417366722 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:48:50.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.253+0000 7fd80f400700 1 --2- 172.21.15.27:0/1417366722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd810070890 0x7fd81010fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:50.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.253+0000 7fd80ea00700 1 -- 172.21.15.27:0/1417366722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd810070890 msgr2=0x7fd81010fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:50.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.253+0000 7fd80ea00700 1 --2- 172.21.15.27:0/1417366722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd810070890 0x7fd81010fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:50.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.254+0000 7fd80ea00700 1 -- 172.21.15.27:0/1417366722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd800005040 con 0x7fd810071e40 2024-04-16T05:48:50.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.254+0000 7fd80ea00700 1 --2- 172.21.15.27:0/1417366722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd810071e40 0x7fd810110250 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc00c810 tx=0x7fd7fc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:50.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.254+0000 7fd807400700 1 -- 172.21.15.27:0/1417366722 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7fc012810 con 0x7fd810071e40 2024-04-16T05:48:50.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.254+0000 7fd807400700 1 -- 172.21.15.27:0/1417366722 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd7fc012e40 con 0x7fd810071e40 2024-04-16T05:48:50.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.254+0000 7fd816400700 1 -- 172.21.15.27:0/1417366722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd810111270 con 0x7fd810071e40 2024-04-16T05:48:50.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.254+0000 7fd807400700 1 -- 172.21.15.27:0/1417366722 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7fc00a370 con 0x7fd810071e40 2024-04-16T05:48:50.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.255+0000 7fd816400700 1 -- 172.21.15.27:0/1417366722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd810111830 con 0x7fd810071e40 2024-04-16T05:48:50.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.256+0000 7fd816400700 1 -- 172.21.15.27:0/1417366722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd81005f5f0 con 0x7fd810071e40 2024-04-16T05:48:50.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.257+0000 7fd807400700 1 -- 172.21.15.27:0/1417366722 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd7fc011810 con 0x7fd810071e40 2024-04-16T05:48:50.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.259+0000 7fd807400700 1 --2- 172.21.15.27:0/1417366722 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd7f406ec10 0x7fd7f40710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:50.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.259+0000 7fd807400700 1 -- 172.21.15.27:0/1417366722 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6736+0+0 (secure 0 0 0) 0x7fd7fc08ed90 con 0x7fd810071e40 2024-04-16T05:48:50.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.259+0000 7fd80f400700 1 --2- 172.21.15.27:0/1417366722 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd7f406ec10 0x7fd7f40710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:50.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.260+0000 7fd80f400700 1 --2- 172.21.15.27:0/1417366722 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd7f406ec10 0x7fd7f40710c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd80000b3b0 tx=0x7fd800009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:50.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.264+0000 7fd807400700 1 -- 172.21.15.27:0/1417366722 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd7fc05b580 con 0x7fd810071e40 2024-04-16T05:48:50.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.611+0000 7fd816400700 1 -- 172.21.15.27:0/1417366722 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd81004bed0 con 0x7fd810071e40 2024-04-16T05:48:50.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.612+0000 7fd807400700 1 -- 172.21.15.27:0/1417366722 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7fd7fc05b110 con 0x7fd810071e40 2024-04-16T05:48:50.612 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:48:50.612 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:48:50.612 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:48:50.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:48:50.612 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:48:50.612 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:48:50.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:48:50.612 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:48:50.612 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 3, 2024-04-16T05:48:50.612 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 3 2024-04-16T05:48:50.612 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:48:50.612 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:48:50.612 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T05:48:50.613 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:48:50.613 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:48:50.613 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 7, 2024-04-16T05:48:50.613 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 7 2024-04-16T05:48:50.613 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:48:50.613 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:48:50.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.616+0000 7fd816400700 1 -- 172.21.15.27:0/1417366722 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd7f406ec10 msgr2=0x7fd7f40710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:50.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.616+0000 7fd816400700 1 --2- 172.21.15.27:0/1417366722 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd7f406ec10 0x7fd7f40710c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd80000b3b0 tx=0x7fd800009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.616+0000 7fd816400700 1 -- 172.21.15.27:0/1417366722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd810071e40 msgr2=0x7fd810110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.616+0000 7fd816400700 1 --2- 172.21.15.27:0/1417366722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd810071e40 0x7fd810110250 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc00c810 tx=0x7fd7fc004470 comp rx=0 tx=0).stop 2024-04-16T05:48:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.616+0000 7fd816400700 1 -- 172.21.15.27:0/1417366722 shutdown_connections 2024-04-16T05:48:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.616+0000 7fd816400700 1 --2- 172.21.15.27:0/1417366722 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd7f406ec10 0x7fd7f40710c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.616+0000 7fd816400700 1 --2- 172.21.15.27:0/1417366722 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd810070890 0x7fd81010fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.616+0000 7fd816400700 1 --2- 172.21.15.27:0/1417366722 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd810071e40 0x7fd810110250 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.617+0000 7fd816400700 1 -- 172.21.15.27:0/1417366722 >> 172.21.15.27:0/1417366722 conn(0x7fd8100f8590 msgr2=0x7fd8100f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.617+0000 7fd816400700 1 -- 172.21.15.27:0/1417366722 shutdown_connections 2024-04-16T05:48:50.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.617+0000 7fd816400700 1 -- 172.21.15.27:0/1417366722 wait complete. 2024-04-16T05:48:50.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:50 smithi122 ceph-mon[80930]: pgmap v122: 65 pgs: 65 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:50.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:50 smithi122 ceph-mon[80930]: from='client.34308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:48:50.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.781+0000 7fb91a400700 1 -- 172.21.15.27:0/752809048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb914103270 msgr2=0x7fb914103680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:50.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.781+0000 7fb91a400700 1 --2- 172.21.15.27:0/752809048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb914103270 0x7fb914103680 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb908004940 tx=0x7fb908009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:50.780 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:50 smithi027 ceph-mon[93635]: pgmap v122: 65 pgs: 65 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:50.780 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:50 smithi027 ceph-mon[93635]: from='client.34308 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:48:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.781+0000 7fb91a400700 1 -- 172.21.15.27:0/752809048 shutdown_connections 2024-04-16T05:48:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.781+0000 7fb91a400700 1 --2- 172.21.15.27:0/752809048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb914104520 0x7fb914104990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.781+0000 7fb91a400700 1 --2- 172.21.15.27:0/752809048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb914103270 0x7fb914103680 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.781+0000 7fb91a400700 1 -- 172.21.15.27:0/752809048 >> 172.21.15.27:0/752809048 conn(0x7fb914072530 msgr2=0x7fb914074940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:50.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.781+0000 7fb91a400700 1 -- 172.21.15.27:0/752809048 shutdown_connections 2024-04-16T05:48:50.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.781+0000 7fb91a400700 1 -- 172.21.15.27:0/752809048 wait complete. 2024-04-16T05:48:50.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.782+0000 7fb91a400700 1 Processor -- start 2024-04-16T05:48:50.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.782+0000 7fb91a400700 1 -- start start 2024-04-16T05:48:50.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.783+0000 7fb91a400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb914103270 0x7fb914113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:50.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.783+0000 7fb91a400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb914104520 0x7fb914114460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:50.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.783+0000 7fb91a400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb914114a60 con 0x7fb914104520 2024-04-16T05:48:50.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.783+0000 7fb91a400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9141151a0 con 0x7fb914103270 2024-04-16T05:48:50.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.783+0000 7fb913e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb914104520 0x7fb914114460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:50.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.783+0000 7fb913e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb914104520 0x7fb914114460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44648/0 (socket says 172.21.15.27:44648) 2024-04-16T05:48:50.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.783+0000 7fb913e00700 1 -- 172.21.15.27:0/3718892582 learned_addr learned my addr 172.21.15.27:0/3718892582 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:48:50.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.783+0000 7fb919000700 1 --2- 172.21.15.27:0/3718892582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb914103270 0x7fb914113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:50.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.783+0000 7fb913e00700 1 -- 172.21.15.27:0/3718892582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb914103270 msgr2=0x7fb914113f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:50.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.783+0000 7fb913e00700 1 --2- 172.21.15.27:0/3718892582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb914103270 0x7fb914113f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:50.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.783+0000 7fb913e00700 1 -- 172.21.15.27:0/3718892582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb908005040 con 0x7fb914104520 2024-04-16T05:48:50.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.784+0000 7fb913e00700 1 --2- 172.21.15.27:0/3718892582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb914104520 0x7fb914114460 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb90400c810 tx=0x7fb904004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:50.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.784+0000 7fb910c00700 1 -- 172.21.15.27:0/3718892582 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb904012810 con 0x7fb914104520 2024-04-16T05:48:50.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.784+0000 7fb910c00700 1 -- 172.21.15.27:0/3718892582 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb904012e40 con 0x7fb914104520 2024-04-16T05:48:50.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.784+0000 7fb91a400700 1 -- 172.21.15.27:0/3718892582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb914115480 con 0x7fb914104520 2024-04-16T05:48:50.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.784+0000 7fb910c00700 1 -- 172.21.15.27:0/3718892582 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb90400a370 con 0x7fb914104520 2024-04-16T05:48:50.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.784+0000 7fb91a400700 1 -- 172.21.15.27:0/3718892582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb914115a40 con 0x7fb914104520 2024-04-16T05:48:50.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.785+0000 7fb91a400700 1 -- 172.21.15.27:0/3718892582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9140760a0 con 0x7fb914104520 2024-04-16T05:48:50.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.787+0000 7fb910c00700 1 -- 172.21.15.27:0/3718892582 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb904011810 con 0x7fb914104520 2024-04-16T05:48:50.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.788+0000 7fb910c00700 1 --2- 172.21.15.27:0/3718892582 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb8fc06ec00 0x7fb8fc0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:50.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.788+0000 7fb910c00700 1 -- 172.21.15.27:0/3718892582 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6736+0+0 (secure 0 0 0) 0x7fb90408ee80 con 0x7fb914104520 2024-04-16T05:48:50.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.788+0000 7fb919000700 1 --2- 172.21.15.27:0/3718892582 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb8fc06ec00 0x7fb8fc0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:50.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.789+0000 7fb919000700 1 --2- 172.21.15.27:0/3718892582 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb8fc06ec00 0x7fb8fc0710b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb908004f90 tx=0x7fb908009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:50.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:50.793+0000 7fb910c00700 1 -- 172.21.15.27:0/3718892582 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb90405b5f0 con 0x7fb914104520 2024-04-16T05:48:51.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.075+0000 7fb91a400700 1 -- 172.21.15.27:0/3718892582 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb91404bed0 con 0x7fb914104520 2024-04-16T05:48:51.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.076+0000 7fb910c00700 1 -- 172.21.15.27:0/3718892582 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1880 (secure 0 0 0) 0x7fb90405b180 con 0x7fb914104520 2024-04-16T05:48:51.076 INFO:teuthology.orchestra.run.smithi027.stdout:e16 2024-04-16T05:48:51.076 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:48:51.076 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:48:51.076 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:48:51.076 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:48:51.076 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:48:51.076 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:48:51.077 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 15 2024-04-16T05:48:51.077 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:48:51.077 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:48:51.077 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:40:32.701649+0000 2024-04-16T05:48:51.077 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:48:51.077 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:48:51.077 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:48:51.077 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:48:51.077 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:48:51.077 INFO:teuthology.orchestra.run.smithi027.stdout:max_xattr_size 65536 2024-04-16T05:48:51.077 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:48:51.077 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:48:51.077 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 46 2024-04-16T05:48:51.077 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:48:51.077 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:48:51.077 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:48:51.078 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=14462} 2024-04-16T05:48:51.078 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:48:51.078 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:48:51.078 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:48:51.078 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:48:51.078 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:48:51.078 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:48:51.078 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:48:51.078 INFO:teuthology.orchestra.run.smithi027.stdout:bal_rank_mask -1 2024-04-16T05:48:51.078 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:48:51.078 INFO:teuthology.orchestra.run.smithi027.stdout:qdb_cluster leader: 0 members: 2024-04-16T05:48:51.078 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:48:51.078 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:48:51.079 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:48:51.079 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:48:51.079 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:48:51.079 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{-1:14474} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2065690722,v1:172.21.15.27:6827/2065690722] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:48:51.079 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:24263} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:48:51.079 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:24273} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:48:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.080+0000 7fb91a400700 1 -- 172.21.15.27:0/3718892582 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb8fc06ec00 msgr2=0x7fb8fc0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.080+0000 7fb91a400700 1 --2- 172.21.15.27:0/3718892582 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb8fc06ec00 0x7fb8fc0710b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb908004f90 tx=0x7fb908009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.080+0000 7fb91a400700 1 -- 172.21.15.27:0/3718892582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb914104520 msgr2=0x7fb914114460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.080+0000 7fb91a400700 1 --2- 172.21.15.27:0/3718892582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb914104520 0x7fb914114460 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb90400c810 tx=0x7fb904004470 comp rx=0 tx=0).stop 2024-04-16T05:48:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.080+0000 7fb91a400700 1 -- 172.21.15.27:0/3718892582 shutdown_connections 2024-04-16T05:48:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.080+0000 7fb91a400700 1 --2- 172.21.15.27:0/3718892582 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb8fc06ec00 0x7fb8fc0710b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.080+0000 7fb91a400700 1 --2- 172.21.15.27:0/3718892582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb914103270 0x7fb914113f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.080+0000 7fb91a400700 1 --2- 172.21.15.27:0/3718892582 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb914104520 0x7fb914114460 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.080+0000 7fb91a400700 1 -- 172.21.15.27:0/3718892582 >> 172.21.15.27:0/3718892582 conn(0x7fb914072530 msgr2=0x7fb914074ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.081+0000 7fb91a400700 1 -- 172.21.15.27:0/3718892582 shutdown_connections 2024-04-16T05:48:51.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.081+0000 7fb91a400700 1 -- 172.21.15.27:0/3718892582 wait complete. 2024-04-16T05:48:51.081 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2024-04-16T05:48:51.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.243+0000 7f5a83200700 1 -- 172.21.15.27:0/173654470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a7c0ffd40 msgr2=0x7f5a7c102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:51.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.243+0000 7f5a83200700 1 --2- 172.21.15.27:0/173654470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a7c0ffd40 0x7f5a7c102160 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c004940 tx=0x7f5a6c009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:51.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.243+0000 7f5a83200700 1 -- 172.21.15.27:0/173654470 shutdown_connections 2024-04-16T05:48:51.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.243+0000 7f5a83200700 1 --2- 172.21.15.27:0/173654470 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a7c0ffd40 0x7f5a7c102160 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:51.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.243+0000 7f5a83200700 1 --2- 172.21.15.27:0/173654470 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a7c0662f0 0x7f5a7c0ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:51.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.243+0000 7f5a83200700 1 -- 172.21.15.27:0/173654470 >> 172.21.15.27:0/173654470 conn(0x7f5a7c0f7270 msgr2=0x7f5a7c0f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:51.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.243+0000 7f5a83200700 1 -- 172.21.15.27:0/173654470 shutdown_connections 2024-04-16T05:48:51.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.243+0000 7f5a83200700 1 -- 172.21.15.27:0/173654470 wait complete. 2024-04-16T05:48:51.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.244+0000 7f5a83200700 1 Processor -- start 2024-04-16T05:48:51.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.244+0000 7f5a83200700 1 -- start start 2024-04-16T05:48:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.245+0000 7f5a83200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a7c0662f0 0x7f5a7c0fd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.245+0000 7f5a83200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a7c0ffd40 0x7f5a7c0fb700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.245+0000 7f5a83200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c0fbcb0 con 0x7f5a7c0ffd40 2024-04-16T05:48:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.245+0000 7f5a83200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c0fbe20 con 0x7f5a7c0662f0 2024-04-16T05:48:51.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.245+0000 7f5a81400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a7c0ffd40 0x7f5a7c0fb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:51.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.245+0000 7f5a81400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a7c0ffd40 0x7f5a7c0fb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44672/0 (socket says 172.21.15.27:44672) 2024-04-16T05:48:51.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.245+0000 7f5a81400700 1 -- 172.21.15.27:0/4293576620 learned_addr learned my addr 172.21.15.27:0/4293576620 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:48:51.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.245+0000 7f5a81e00700 1 --2- 172.21.15.27:0/4293576620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a7c0662f0 0x7f5a7c0fd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:51.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.246+0000 7f5a81400700 1 -- 172.21.15.27:0/4293576620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a7c0662f0 msgr2=0x7f5a7c0fd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:51.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.246+0000 7f5a81400700 1 --2- 172.21.15.27:0/4293576620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a7c0662f0 0x7f5a7c0fd230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:51.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.246+0000 7f5a81400700 1 -- 172.21.15.27:0/4293576620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6c005040 con 0x7f5a7c0ffd40 2024-04-16T05:48:51.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.246+0000 7f5a81e00700 1 --2- 172.21.15.27:0/4293576620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a7c0662f0 0x7f5a7c0fd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T05:48:51.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.246+0000 7f5a81400700 1 --2- 172.21.15.27:0/4293576620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a7c0ffd40 0x7f5a7c0fb700 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c009150 tx=0x7f5a6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:51.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.246+0000 7f5a76000700 1 -- 172.21.15.27:0/4293576620 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a6c00e040 con 0x7f5a7c0ffd40 2024-04-16T05:48:51.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.247+0000 7f5a76000700 1 -- 172.21.15.27:0/4293576620 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a6c00bbf0 con 0x7f5a7c0ffd40 2024-04-16T05:48:51.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.247+0000 7f5a83200700 1 -- 172.21.15.27:0/4293576620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a7c0fc0a0 con 0x7f5a7c0ffd40 2024-04-16T05:48:51.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.247+0000 7f5a76000700 1 -- 172.21.15.27:0/4293576620 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5a6c01c630 con 0x7f5a7c0ffd40 2024-04-16T05:48:51.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.247+0000 7f5a83200700 1 -- 172.21.15.27:0/4293576620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a7c1158d0 con 0x7f5a7c0ffd40 2024-04-16T05:48:51.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.248+0000 7f5a83200700 1 -- 172.21.15.27:0/4293576620 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a7c05f5f0 con 0x7f5a7c0ffd40 2024-04-16T05:48:51.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.249+0000 7f5a76000700 1 -- 172.21.15.27:0/4293576620 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5a6c00bd60 con 0x7f5a7c0ffd40 2024-04-16T05:48:51.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.250+0000 7f5a76000700 1 --2- 172.21.15.27:0/4293576620 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5a6406ec10 0x7f5a640710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:51.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.250+0000 7f5a76000700 1 -- 172.21.15.27:0/4293576620 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f5a6c08fd40 con 0x7f5a7c0ffd40 2024-04-16T05:48:51.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.251+0000 7f5a81e00700 1 --2- 172.21.15.27:0/4293576620 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5a6406ec10 0x7f5a640710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:51.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.251+0000 7f5a81e00700 1 --2- 172.21.15.27:0/4293576620 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5a6406ec10 0x7f5a640710c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5a70009fd0 tx=0x7f5a70009f00 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:51.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.255+0000 7f5a76000700 1 -- 172.21.15.27:0/4293576620 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5a6c058a80 con 0x7f5a7c0ffd40 2024-04-16T05:48:51.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.513+0000 7f5a83200700 1 -- 172.21.15.27:0/4293576620 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a7c0f8db0 con 0x7f5a6406ec10 2024-04-16T05:48:51.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.515+0000 7f5a76000700 1 -- 172.21.15.27:0/4293576620 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+400 (secure 0 0 0) 0x7f5a7c0f8db0 con 0x7f5a6406ec10 2024-04-16T05:48:51.515 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:48:51.515 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T05:48:51.515 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:48:51.515 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:48:51.515 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [ 2024-04-16T05:48:51.515 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr", 2024-04-16T05:48:51.515 INFO:teuthology.orchestra.run.smithi027.stdout: "mon", 2024-04-16T05:48:51.515 INFO:teuthology.orchestra.run.smithi027.stdout: "crash" 2024-04-16T05:48:51.515 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-16T05:48:51.515 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "9/21 daemons upgraded", 2024-04-16T05:48:51.515 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Currently upgrading osd daemons", 2024-04-16T05:48:51.515 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:48:51.515 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:48:51.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.518+0000 7f5a83200700 1 -- 172.21.15.27:0/4293576620 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5a6406ec10 msgr2=0x7f5a640710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:51.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.518+0000 7f5a83200700 1 --2- 172.21.15.27:0/4293576620 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5a6406ec10 0x7f5a640710c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5a70009fd0 tx=0x7f5a70009f00 comp rx=0 tx=0).stop 2024-04-16T05:48:51.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.518+0000 7f5a83200700 1 -- 172.21.15.27:0/4293576620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a7c0ffd40 msgr2=0x7f5a7c0fb700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:51.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.519+0000 7f5a83200700 1 --2- 172.21.15.27:0/4293576620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a7c0ffd40 0x7f5a7c0fb700 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c009150 tx=0x7f5a6c009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:51.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.519+0000 7f5a83200700 1 -- 172.21.15.27:0/4293576620 shutdown_connections 2024-04-16T05:48:51.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.519+0000 7f5a83200700 1 --2- 172.21.15.27:0/4293576620 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5a6406ec10 0x7f5a640710c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:51.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.519+0000 7f5a83200700 1 --2- 172.21.15.27:0/4293576620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a7c0662f0 0x7f5a7c0fd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:51.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.519+0000 7f5a83200700 1 --2- 172.21.15.27:0/4293576620 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a7c0ffd40 0x7f5a7c0fb700 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:51.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.519+0000 7f5a83200700 1 -- 172.21.15.27:0/4293576620 >> 172.21.15.27:0/4293576620 conn(0x7f5a7c0f7270 msgr2=0x7f5a7c0f7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:51.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.519+0000 7f5a83200700 1 -- 172.21.15.27:0/4293576620 shutdown_connections 2024-04-16T05:48:51.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.519+0000 7f5a83200700 1 -- 172.21.15.27:0/4293576620 wait complete. 2024-04-16T05:48:51.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.675+0000 7f2cce400700 1 -- 172.21.15.27:0/3651641314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc81046b0 msgr2=0x7f2cc8104b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:51.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.675+0000 7f2cce400700 1 --2- 172.21.15.27:0/3651641314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc81046b0 0x7f2cc8104b20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4004940 tx=0x7f2cb4009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:51.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.675+0000 7f2cce400700 1 -- 172.21.15.27:0/3651641314 shutdown_connections 2024-04-16T05:48:51.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.675+0000 7f2cce400700 1 --2- 172.21.15.27:0/3651641314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc81046b0 0x7f2cc8104b20 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:51.674 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.675+0000 7f2cce400700 1 --2- 172.21.15.27:0/3651641314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cc8103400 0x7f2cc8103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:51.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.675+0000 7f2cce400700 1 -- 172.21.15.27:0/3651641314 >> 172.21.15.27:0/3651641314 conn(0x7f2cc8072590 msgr2=0x7f2cc80749c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:51.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.675+0000 7f2cce400700 1 -- 172.21.15.27:0/3651641314 shutdown_connections 2024-04-16T05:48:51.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.675+0000 7f2cce400700 1 -- 172.21.15.27:0/3651641314 wait complete. 2024-04-16T05:48:51.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.676+0000 7f2cce400700 1 Processor -- start 2024-04-16T05:48:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.676+0000 7f2cce400700 1 -- start start 2024-04-16T05:48:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.676+0000 7f2cce400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc8103400 0x7f2cc8114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.676+0000 7f2cce400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cc81046b0 0x7f2cc8114550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.676+0000 7f2cce400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc8114b50 con 0x7f2cc8103400 2024-04-16T05:48:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.676+0000 7f2cce400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc81153a0 con 0x7f2cc81046b0 2024-04-16T05:48:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.677+0000 7f2cc7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc8103400 0x7f2cc8114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.677+0000 7f2cc7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc8103400 0x7f2cc8114010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44700/0 (socket says 172.21.15.27:44700) 2024-04-16T05:48:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.677+0000 7f2cc7400700 1 -- 172.21.15.27:0/2730707288 learned_addr learned my addr 172.21.15.27:0/2730707288 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:48:51.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.677+0000 7f2cc6a00700 1 --2- 172.21.15.27:0/2730707288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cc81046b0 0x7f2cc8114550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:51.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.677+0000 7f2cc7400700 1 -- 172.21.15.27:0/2730707288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cc81046b0 msgr2=0x7f2cc8114550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:51.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.677+0000 7f2cc7400700 1 --2- 172.21.15.27:0/2730707288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cc81046b0 0x7f2cc8114550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:51.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.677+0000 7f2cc7400700 1 -- 172.21.15.27:0/2730707288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb4005040 con 0x7f2cc8103400 2024-04-16T05:48:51.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.678+0000 7f2cc7400700 1 --2- 172.21.15.27:0/2730707288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc8103400 0x7f2cc8114010 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8002a50 tx=0x7f2cb80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:51.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.678+0000 7f2cbf400700 1 -- 172.21.15.27:0/2730707288 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cb8007cc0 con 0x7f2cc8103400 2024-04-16T05:48:51.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.678+0000 7f2cbf400700 1 -- 172.21.15.27:0/2730707288 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2cb800f040 con 0x7f2cc8103400 2024-04-16T05:48:51.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.678+0000 7f2cce400700 1 -- 172.21.15.27:0/2730707288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc8115680 con 0x7f2cc8103400 2024-04-16T05:48:51.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.678+0000 7f2cbf400700 1 -- 172.21.15.27:0/2730707288 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2cb800da30 con 0x7f2cc8103400 2024-04-16T05:48:51.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.678+0000 7f2cce400700 1 -- 172.21.15.27:0/2730707288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc8115bc0 con 0x7f2cc8103400 2024-04-16T05:48:51.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.679+0000 7f2cce400700 1 -- 172.21.15.27:0/2730707288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc805f5f0 con 0x7f2cc8103400 2024-04-16T05:48:51.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.680+0000 7f2cbf400700 1 -- 172.21.15.27:0/2730707288 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2cb800db90 con 0x7f2cc8103400 2024-04-16T05:48:51.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.682+0000 7f2cbf400700 1 --2- 172.21.15.27:0/2730707288 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2cac06ec10 0x7f2cac0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:48:51.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.682+0000 7f2cbf400700 1 -- 172.21.15.27:0/2730707288 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f2cb808e6c0 con 0x7f2cc8103400 2024-04-16T05:48:51.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.682+0000 7f2cc6a00700 1 --2- 172.21.15.27:0/2730707288 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2cac06ec10 0x7f2cac0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:48:51.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.683+0000 7f2cc6a00700 1 --2- 172.21.15.27:0/2730707288 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2cac06ec10 0x7f2cac0710c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2cb400b3b0 tx=0x7f2cb4009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:48:51.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:51.687+0000 7f2cbf400700 1 -- 172.21.15.27:0/2730707288 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2cb805aeb0 con 0x7f2cc8103400 2024-04-16T05:48:51.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:51 smithi027 ceph-mon[93635]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:48:51.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:51 smithi027 ceph-mon[93635]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:48:51.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:51 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1417366722' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:48:51.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:51 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/3718892582' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:48:51.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:51 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:51.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:51 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:51.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:51 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:48:51.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:51 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:48:51.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:51 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:52.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:52.005+0000 7f2cce400700 1 -- 172.21.15.27:0/2730707288 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2cc8115f90 con 0x7f2cc8103400 2024-04-16T05:48:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:52.006+0000 7f2cbf400700 1 -- 172.21.15.27:0/2730707288 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f2cb805aa40 con 0x7f2cc8103400 2024-04-16T05:48:52.005 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:48:52.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:52.009+0000 7f2cce400700 1 -- 172.21.15.27:0/2730707288 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2cac06ec10 msgr2=0x7f2cac0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:52.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:52.009+0000 7f2cce400700 1 --2- 172.21.15.27:0/2730707288 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2cac06ec10 0x7f2cac0710c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2cb400b3b0 tx=0x7f2cb4009e30 comp rx=0 tx=0).stop 2024-04-16T05:48:52.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:52.009+0000 7f2cce400700 1 -- 172.21.15.27:0/2730707288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc8103400 msgr2=0x7f2cc8114010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:48:52.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:52.009+0000 7f2cce400700 1 --2- 172.21.15.27:0/2730707288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc8103400 0x7f2cc8114010 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8002a50 tx=0x7f2cb80046b0 comp rx=0 tx=0).stop 2024-04-16T05:48:52.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:52.009+0000 7f2cce400700 1 -- 172.21.15.27:0/2730707288 shutdown_connections 2024-04-16T05:48:52.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:52.009+0000 7f2cce400700 1 --2- 172.21.15.27:0/2730707288 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2cac06ec10 0x7f2cac0710c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:52.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:52.009+0000 7f2cce400700 1 --2- 172.21.15.27:0/2730707288 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2cc8103400 0x7f2cc8114010 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:52.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:52.009+0000 7f2cce400700 1 --2- 172.21.15.27:0/2730707288 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2cc81046b0 0x7f2cc8114550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:48:52.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:52.009+0000 7f2cce400700 1 -- 172.21.15.27:0/2730707288 >> 172.21.15.27:0/2730707288 conn(0x7f2cc8072590 msgr2=0x7f2cc8073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:48:52.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:52.009+0000 7f2cce400700 1 -- 172.21.15.27:0/2730707288 shutdown_connections 2024-04-16T05:48:52.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:48:52.009+0000 7f2cce400700 1 -- 172.21.15.27:0/2730707288 wait complete. 2024-04-16T05:48:52.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:51 smithi122 ceph-mon[80930]: from='client.34312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:48:52.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:51 smithi122 ceph-mon[80930]: from='client.34316 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:48:52.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:51 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1417366722' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:48:52.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:51 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/3718892582' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:48:52.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:51 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:52.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:51 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:52.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:51 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:48:52.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:51 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:48:52.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:51 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:52.726 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:52 smithi122 ceph-mon[80930]: pgmap v123: 65 pgs: 65 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:52.727 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:52 smithi122 ceph-mon[80930]: from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:48:52.727 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:52 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:48:52.727 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:52 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:48:52.727 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:52 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:48:52.727 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:52 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:48:52.727 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:52 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-16T05:48:52.727 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:52 smithi122 ceph-mon[80930]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-16T05:48:52.727 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:52 smithi122 ceph-mon[80930]: Upgrade: osd.3 is safe to restart 2024-04-16T05:48:52.727 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:52 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/2730707288' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:48:52.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:52 smithi027 ceph-mon[93635]: pgmap v123: 65 pgs: 65 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:52.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:52 smithi027 ceph-mon[93635]: from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:48:52.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:52 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:48:52.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:52 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:48:52.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:52 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:48:52.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:52 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:48:52.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:52 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-16T05:48:52.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:52 smithi027 ceph-mon[93635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-16T05:48:52.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:52 smithi027 ceph-mon[93635]: Upgrade: osd.3 is safe to restart 2024-04-16T05:48:52.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:52 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/2730707288' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:48:54.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:54 smithi027 ceph-mon[93635]: pgmap v124: 65 pgs: 65 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:54.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:54 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:54.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:54 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T05:48:54.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:54 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:48:54.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:54 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:54.962 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:54 smithi122 ceph-mon[80930]: pgmap v124: 65 pgs: 65 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:54.962 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:54 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:54.962 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:54 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-16T05:48:54.962 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:54 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:48:54.962 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:54 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:48:55.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:55 smithi027 ceph-mon[93635]: Upgrade: Updating osd.3 2024-04-16T05:48:55.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:55 smithi027 ceph-mon[93635]: Deploying daemon osd.3 on smithi122 2024-04-16T05:48:56.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:55 smithi122 ceph-mon[80930]: Upgrade: Updating osd.3 2024-04-16T05:48:56.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:55 smithi122 ceph-mon[80930]: Deploying daemon osd.3 on smithi122 2024-04-16T05:48:56.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:56 smithi122 ceph-mon[80930]: pgmap v125: 65 pgs: 65 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:56.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:56 smithi027 ceph-mon[93635]: pgmap v125: 65 pgs: 65 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:58.746 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:48:58 smithi122 ceph-mon[80930]: pgmap v126: 65 pgs: 65 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:58.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:48:58 smithi027 ceph-mon[93635]: pgmap v126: 65 pgs: 65 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail 2024-04-16T05:48:59.021 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:48:58 smithi122 systemd[1]: Stopping Ceph osd.3 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T05:48:59.772 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:48:59 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3[41550]: 2024-04-16T05:48:59.309+0000 7fd53e400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T05:48:59.772 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:48:59 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3[41550]: 2024-04-16T05:48:59.309+0000 7fd53e400700 -1 osd.3 68 *** Got signal Terminated *** 2024-04-16T05:48:59.772 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:48:59 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3[41550]: 2024-04-16T05:48:59.309+0000 7fd53e400700 -1 osd.3 68 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T05:49:00.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:00 smithi122 ceph-mon[80930]: pgmap v127: 65 pgs: 65 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail 2024-04-16T05:49:00.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:00.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:49:00.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:00 smithi122 ceph-mon[80930]: osd.3 marked itself down and dead 2024-04-16T05:49:00.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:00 smithi027 ceph-mon[93635]: pgmap v127: 65 pgs: 65 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail 2024-04-16T05:49:00.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:00.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:49:00.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:00 smithi027 ceph-mon[93635]: osd.3 marked itself down and dead 2024-04-16T05:49:00.731 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:00 smithi122 podman[89721]: 2024-04-16 05:49:00.411553423 +0000 UTC m=+1.377637160 container died d6724ad05f9fc69d77d5c9eb2541bcd5b750154b4eb4dc49e7e28fd0dc2d34ae (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3, ceph=True, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-64502-quincy-3-45317f9, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-16T05:49:01.023 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:00 smithi122 podman[89721]: 2024-04-16 05:49:00.877437217 +0000 UTC m=+1.843521024 container cleanup d6724ad05f9fc69d77d5c9eb2541bcd5b750154b4eb4dc49e7e28fd0dc2d34ae (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0) 2024-04-16T05:49:01.023 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:00 smithi122 bash[89721]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3 2024-04-16T05:49:01.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:01 smithi027 ceph-mon[93635]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T05:49:01.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:01 smithi027 ceph-mon[93635]: osdmap e69: 6 total, 5 up, 6 in 2024-04-16T05:49:01.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:01 smithi027 ceph-mon[93635]: osdmap e70: 6 total, 5 up, 6 in 2024-04-16T05:49:01.521 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:01 smithi122 podman[89739]: 2024-04-16 05:49:01.178438502 +0000 UTC m=+0.759991249 container remove d6724ad05f9fc69d77d5c9eb2541bcd5b750154b4eb4dc49e7e28fd0dc2d34ae (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-64502-quincy-3-45317f9, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T05:49:01.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:01 smithi122 ceph-mon[80930]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T05:49:01.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:01 smithi122 ceph-mon[80930]: osdmap e69: 6 total, 5 up, 6 in 2024-04-16T05:49:01.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:01 smithi122 ceph-mon[80930]: osdmap e70: 6 total, 5 up, 6 in 2024-04-16T05:49:02.043 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:01 smithi122 podman[89825]: 2024-04-16 05:49:01.69405524 +0000 UTC m=+0.032911716 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T05:49:02.043 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:01 smithi122 podman[89825]: 2024-04-16 05:49:01.828721933 +0000 UTC m=+0.167578431 container create 662d0ba1a63627734d62b61fe8285b5c9db46aa724adef5165829aed2e210c15 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T05:49:02.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:02 smithi027 ceph-mon[93635]: pgmap v129: 65 pgs: 3 peering, 13 stale+active+clean, 49 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail 2024-04-16T05:49:02.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:02 smithi122 ceph-mon[80930]: pgmap v129: 65 pgs: 3 peering, 13 stale+active+clean, 49 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail 2024-04-16T05:49:02.521 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:02 smithi122 podman[89825]: 2024-04-16 05:49:02.091950964 +0000 UTC m=+0.430807491 container init 662d0ba1a63627734d62b61fe8285b5c9db46aa724adef5165829aed2e210c15 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-16T05:49:02.521 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:02 smithi122 podman[89825]: 2024-04-16 05:49:02.10357212 +0000 UTC m=+0.442428637 container start 662d0ba1a63627734d62b61fe8285b5c9db46aa724adef5165829aed2e210c15 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-16T05:49:02.521 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:02 smithi122 podman[89825]: 2024-04-16 05:49:02.190393304 +0000 UTC m=+0.529249787 container attach 662d0ba1a63627734d62b61fe8285b5c9db46aa724adef5165829aed2e210c15 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS) 2024-04-16T05:49:02.946 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:02 smithi122 podman[89825]: 2024-04-16 05:49:02.557348493 +0000 UTC m=+0.896204972 container died 662d0ba1a63627734d62b61fe8285b5c9db46aa724adef5165829aed2e210c15 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T05:49:03.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:03 smithi122 ceph-mon[80930]: Health check failed: Degraded data redundancy: 4/72 objects degraded (5.556%), 2 pgs degraded (PG_DEGRADED) 2024-04-16T05:49:03.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:03 smithi027 ceph-mon[93635]: Health check failed: Degraded data redundancy: 4/72 objects degraded (5.556%), 2 pgs degraded (PG_DEGRADED) 2024-04-16T05:49:03.679 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:03 smithi122 podman[89851]: 2024-04-16 05:49:03.28539279 +0000 UTC m=+0.707032937 container remove 662d0ba1a63627734d62b61fe8285b5c9db46aa724adef5165829aed2e210c15 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409) 2024-04-16T05:49:03.679 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:03 smithi122 systemd[1]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.3.service: Deactivated successfully. 2024-04-16T05:49:03.679 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:03 smithi122 systemd[1]: Stopped Ceph osd.3 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968. 2024-04-16T05:49:03.679 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:03 smithi122 systemd[1]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.3.service: Consumed 13.482s CPU time. 2024-04-16T05:49:04.021 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:03 smithi122 systemd[1]: Starting Ceph osd.3 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T05:49:04.330 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:04 smithi122 ceph-mon[80930]: pgmap v131: 65 pgs: 2 active+undersized+degraded, 3 peering, 12 stale+active+clean, 3 active+undersized, 45 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail; 4/72 objects degraded (5.556%) 2024-04-16T05:49:04.330 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:04 smithi122 podman[89952]: 2024-04-16 05:49:03.954276171 +0000 UTC m=+0.035576211 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T05:49:04.330 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:04 smithi122 podman[89952]: 2024-04-16 05:49:04.061340716 +0000 UTC m=+0.142640793 container create 9deff42763da6e7a9953e1e04ea1a5c55af705c1de8fe19888dde5a8efb25a73 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-activate, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409) 2024-04-16T05:49:04.330 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:04 smithi122 podman[89952]: 2024-04-16 05:49:04.331753432 +0000 UTC m=+0.413053552 container init 9deff42763da6e7a9953e1e04ea1a5c55af705c1de8fe19888dde5a8efb25a73 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-activate, org.label-schema.build-date=20240409, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-16T05:49:04.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:04 smithi027 ceph-mon[93635]: pgmap v131: 65 pgs: 2 active+undersized+degraded, 3 peering, 12 stale+active+clean, 3 active+undersized, 45 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail; 4/72 objects degraded (5.556%) 2024-04-16T05:49:04.771 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:04 smithi122 podman[89952]: 2024-04-16 05:49:04.344032034 +0000 UTC m=+0.425332137 container start 9deff42763da6e7a9953e1e04ea1a5c55af705c1de8fe19888dde5a8efb25a73 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-activate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-16T05:49:04.772 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:04 smithi122 podman[89952]: 2024-04-16 05:49:04.430516128 +0000 UTC m=+0.511816212 container attach 9deff42763da6e7a9953e1e04ea1a5c55af705c1de8fe19888dde5a8efb25a73 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-activate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-04-16T05:49:05.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 348, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 90042 2024-04-16T05:49:05.069 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 348, success, outsize: 120 2024-04-16T05:49:05.523 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:05 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-activate[89967]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-16T05:49:05.523 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:05 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-activate[89967]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-16T05:49:05.524 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:05 smithi122 bash[89952]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-16T05:49:05.524 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:05 smithi122 bash[89952]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-16T05:49:05.524 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:05 smithi122 bash[89952]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-16T05:49:05.524 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:05 smithi122 bash[89952]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-16T05:49:05.524 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:05 smithi122 bash[89952]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-16T05:49:05.524 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:05 smithi122 bash[89952]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-16T05:49:05.524 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:05 smithi122 bash[89952]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-16T05:49:05.524 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:05 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-activate[89967]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-16T05:49:05.524 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:05 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-activate[89967]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-16T05:49:05.524 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:05 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-activate[89967]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3/block 2024-04-16T05:49:05.524 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:05 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-activate[89967]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-16T05:49:05.524 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:05 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-activate[89967]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-16T05:49:05.525 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:05 smithi122 podman[89952]: 2024-04-16 05:49:05.256448083 +0000 UTC m=+1.337748131 container died 9deff42763da6e7a9953e1e04ea1a5c55af705c1de8fe19888dde5a8efb25a73 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-04-16T05:49:06.264 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:06 smithi122 ceph-mon[80930]: pgmap v132: 65 pgs: 8 active+undersized+degraded, 3 peering, 22 active+undersized, 32 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T05:49:06.264 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:05 smithi122 podman[90086]: 2024-04-16 05:49:05.883400235 +0000 UTC m=+0.605166861 container remove 9deff42763da6e7a9953e1e04ea1a5c55af705c1de8fe19888dde5a8efb25a73 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-activate, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T05:49:06.521 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:06 smithi122 podman[90143]: 2024-04-16 05:49:06.172295888 +0000 UTC m=+0.036712578 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T05:49:06.521 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:06 smithi122 podman[90143]: 2024-04-16 05:49:06.309623092 +0000 UTC m=+0.174039813 container create 897d3579f929bfb3dba5fbb1295b605a9c49f88ee2a61b1fde40b675df1d1b8a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, GIT_CLEAN=True) 2024-04-16T05:49:06.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:06 smithi027 ceph-mon[93635]: pgmap v132: 65 pgs: 8 active+undersized+degraded, 3 peering, 22 active+undersized, 32 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T05:49:07.021 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:06 smithi122 podman[90143]: 2024-04-16 05:49:06.554963711 +0000 UTC m=+0.419380439 container init 897d3579f929bfb3dba5fbb1295b605a9c49f88ee2a61b1fde40b675df1d1b8a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD) 2024-04-16T05:49:07.022 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:06 smithi122 podman[90143]: 2024-04-16 05:49:06.5657037 +0000 UTC m=+0.430120421 container start 897d3579f929bfb3dba5fbb1295b605a9c49f88ee2a61b1fde40b675df1d1b8a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-16T05:49:07.022 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:06 smithi122 ceph-osd[90162]: -- 172.21.15.122:0/3117150750 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5595e4cf1680 con 0x5595e4cbb000 2024-04-16T05:49:07.022 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:06 smithi122 bash[90143]: 897d3579f929bfb3dba5fbb1295b605a9c49f88ee2a61b1fde40b675df1d1b8a 2024-04-16T05:49:07.022 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:06 smithi122 systemd[1]: Started Ceph osd.3 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968. 2024-04-16T05:49:07.521 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:07 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3[90158]: 2024-04-16T05:49:07.151+0000 7f7348b3b7c0 -1 Falling back to public interface 2024-04-16T05:49:08.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:07 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:08.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:07 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:08.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:07 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:49:08.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:07 smithi122 ceph-mon[80930]: pgmap v133: 65 pgs: 9 active+undersized+degraded, 24 active+undersized, 32 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail; 14/72 objects degraded (19.444%) 2024-04-16T05:49:08.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:07 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:08.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:07 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:08.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:07 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:49:08.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:07 smithi027 ceph-mon[93635]: pgmap v133: 65 pgs: 9 active+undersized+degraded, 24 active+undersized, 32 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail; 14/72 objects degraded (19.444%) 2024-04-16T05:49:10.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:10 smithi027 ceph-mon[93635]: pgmap v134: 65 pgs: 9 active+undersized+degraded, 24 active+undersized, 32 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail; 14/72 objects degraded (19.444%) 2024-04-16T05:49:10.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:10 smithi027 ceph-mon[93635]: Health check update: Degraded data redundancy: 14/72 objects degraded (19.444%), 9 pgs degraded (PG_DEGRADED) 2024-04-16T05:49:10.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:10 smithi122 ceph-mon[80930]: pgmap v134: 65 pgs: 9 active+undersized+degraded, 24 active+undersized, 32 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail; 14/72 objects degraded (19.444%) 2024-04-16T05:49:10.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:10 smithi122 ceph-mon[80930]: Health check update: Degraded data redundancy: 14/72 objects degraded (19.444%), 9 pgs degraded (PG_DEGRADED) 2024-04-16T05:49:11.080 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:10 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3[90158]: 2024-04-16T05:49:10.817+0000 7f7348b3b7c0 -1 osd.3 68 log_to_monitors true 2024-04-16T05:49:11.080 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:11 smithi122 ceph-mon[80930]: from='osd.3 [v2:172.21.15.122:6800/3174341439,v1:172.21.15.122:6801/3174341439]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T05:49:11.080 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:11 smithi122 ceph-mon[80930]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T05:49:11.080 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:11 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:11.080 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:11 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:11.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:11 smithi027 ceph-mon[93635]: from='osd.3 [v2:172.21.15.122:6800/3174341439,v1:172.21.15.122:6801/3174341439]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T05:49:11.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:11 smithi027 ceph-mon[93635]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-16T05:49:11.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:11 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:11.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:11 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:11.521 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 05:49:11 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3[90158]: 2024-04-16T05:49:11.080+0000 7f7337e00700 -1 osd.3 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T05:49:12.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:12 smithi122 ceph-mon[80930]: pgmap v135: 65 pgs: 9 active+undersized+degraded, 24 active+undersized, 32 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail; 14/72 objects degraded (19.444%) 2024-04-16T05:49:12.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:12 smithi122 ceph-mon[80930]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-16T05:49:12.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:12 smithi122 ceph-mon[80930]: from='osd.3 [v2:172.21.15.122:6800/3174341439,v1:172.21.15.122:6801/3174341439]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:49:12.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:12 smithi122 ceph-mon[80930]: osdmap e71: 6 total, 5 up, 6 in 2024-04-16T05:49:12.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:12 smithi122 ceph-mon[80930]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:49:12.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:12 smithi027 ceph-mon[93635]: pgmap v135: 65 pgs: 9 active+undersized+degraded, 24 active+undersized, 32 active+clean; 584 KiB data, 958 MiB used, 535 GiB / 536 GiB avail; 14/72 objects degraded (19.444%) 2024-04-16T05:49:12.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:12 smithi027 ceph-mon[93635]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-16T05:49:12.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:12 smithi027 ceph-mon[93635]: from='osd.3 [v2:172.21.15.122:6800/3174341439,v1:172.21.15.122:6801/3174341439]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:49:12.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:12 smithi027 ceph-mon[93635]: osdmap e71: 6 total, 5 up, 6 in 2024-04-16T05:49:12.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:12 smithi027 ceph-mon[93635]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:49:13.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:13 smithi122 ceph-mon[80930]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T05:49:13.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:13 smithi122 ceph-mon[80930]: osd.3 [v2:172.21.15.122:6800/3174341439,v1:172.21.15.122:6801/3174341439] boot 2024-04-16T05:49:13.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:13 smithi122 ceph-mon[80930]: osdmap e72: 6 total, 6 up, 6 in 2024-04-16T05:49:13.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T05:49:13.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:13.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:13.562 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:13 smithi027 ceph-mon[93635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T05:49:13.562 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:13 smithi027 ceph-mon[93635]: osd.3 [v2:172.21.15.122:6800/3174341439,v1:172.21.15.122:6801/3174341439] boot 2024-04-16T05:49:13.562 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:13 smithi027 ceph-mon[93635]: osdmap e72: 6 total, 6 up, 6 in 2024-04-16T05:49:13.563 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-16T05:49:13.563 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:13.563 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:14.224 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:14 smithi122 ceph-mon[80930]: pgmap v138: 65 pgs: 9 active+undersized+degraded, 24 active+undersized, 32 active+clean; 584 KiB data, 694 MiB used, 536 GiB / 536 GiB avail; 14/72 objects degraded (19.444%) 2024-04-16T05:49:14.224 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:14 smithi122 ceph-mon[80930]: osdmap e73: 6 total, 6 up, 6 in 2024-04-16T05:49:14.224 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:14.224 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:49:14.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:14 smithi027 ceph-mon[93635]: pgmap v138: 65 pgs: 9 active+undersized+degraded, 24 active+undersized, 32 active+clean; 584 KiB data, 694 MiB used, 536 GiB / 536 GiB avail; 14/72 objects degraded (19.444%) 2024-04-16T05:49:14.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:14 smithi027 ceph-mon[93635]: osdmap e73: 6 total, 6 up, 6 in 2024-04-16T05:49:14.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:14.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:49:15.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:15 smithi027 ceph-mon[93635]: Health check update: Degraded data redundancy: 12/72 objects degraded (16.667%), 7 pgs degraded (PG_DEGRADED) 2024-04-16T05:49:15.350 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:15 smithi122 ceph-mon[80930]: Health check update: Degraded data redundancy: 12/72 objects degraded (16.667%), 7 pgs degraded (PG_DEGRADED) 2024-04-16T05:49:16.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:16 smithi122 ceph-mon[80930]: pgmap v140: 65 pgs: 7 active+undersized+degraded, 13 active+undersized, 45 active+clean; 584 KiB data, 694 MiB used, 536 GiB / 536 GiB avail; 12/72 objects degraded (16.667%) 2024-04-16T05:49:16.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:16 smithi027 ceph-mon[93635]: pgmap v140: 65 pgs: 7 active+undersized+degraded, 13 active+undersized, 45 active+clean; 584 KiB data, 694 MiB used, 536 GiB / 536 GiB avail; 12/72 objects degraded (16.667%) 2024-04-16T05:49:16.785 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 350, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 91859 2024-04-16T05:49:16.786 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 350, success, outsize: 120 2024-04-16T05:49:16.814 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 352, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 91942 2024-04-16T05:49:16.815 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 352, success, outsize: 120 2024-04-16T05:49:16.837 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 354, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92028 2024-04-16T05:49:16.837 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 354, success, outsize: 120 2024-04-16T05:49:16.859 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 356, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92120 2024-04-16T05:49:16.860 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 356, success, outsize: 120 2024-04-16T05:49:16.882 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 358, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92209 2024-04-16T05:49:16.882 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 358, success, outsize: 120 2024-04-16T05:49:16.905 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 360, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92292 2024-04-16T05:49:16.905 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 360, success, outsize: 120 2024-04-16T05:49:16.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 362, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92378 2024-04-16T05:49:16.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 362, success, outsize: 120 2024-04-16T05:49:16.957 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 364, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92470 2024-04-16T05:49:16.957 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 364, success, outsize: 120 2024-04-16T05:49:16.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 366, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92559 2024-04-16T05:49:16.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 366, success, outsize: 120 2024-04-16T05:49:16.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 368, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92642 2024-04-16T05:49:16.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 368, success, outsize: 120 2024-04-16T05:49:17.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 370, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92728 2024-04-16T05:49:17.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 370, success, outsize: 120 2024-04-16T05:49:17.038 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 372, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92820 2024-04-16T05:49:17.039 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 372, success, outsize: 120 2024-04-16T05:49:17.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 374, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92909 2024-04-16T05:49:17.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 374, success, outsize: 120 2024-04-16T05:49:17.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 376, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 92992 2024-04-16T05:49:17.081 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 376, success, outsize: 120 2024-04-16T05:49:17.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 378, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93078 2024-04-16T05:49:17.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 378, success, outsize: 120 2024-04-16T05:49:17.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 380, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93170 2024-04-16T05:49:17.123 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 380, success, outsize: 120 2024-04-16T05:49:17.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 382, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93259 2024-04-16T05:49:17.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 382, success, outsize: 120 2024-04-16T05:49:17.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 384, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93342 2024-04-16T05:49:17.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 384, success, outsize: 120 2024-04-16T05:49:17.188 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 386, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93428 2024-04-16T05:49:17.189 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 386, success, outsize: 120 2024-04-16T05:49:17.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 388, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93520 2024-04-16T05:49:17.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 388, success, outsize: 120 2024-04-16T05:49:17.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 390, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93609 2024-04-16T05:49:17.233 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 390, success, outsize: 120 2024-04-16T05:49:17.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 392, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93692 2024-04-16T05:49:17.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 392, success, outsize: 120 2024-04-16T05:49:17.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 394, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93778 2024-04-16T05:49:17.274 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 394, success, outsize: 120 2024-04-16T05:49:17.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 396, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93870 2024-04-16T05:49:17.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 396, success, outsize: 120 2024-04-16T05:49:17.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 398, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 93959 2024-04-16T05:49:17.316 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 398, success, outsize: 120 2024-04-16T05:49:17.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 400, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 94042 2024-04-16T05:49:17.336 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 400, success, outsize: 120 2024-04-16T05:49:17.357 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 402, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 94128 2024-04-16T05:49:17.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 402, success, outsize: 120 2024-04-16T05:49:17.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 404, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 94220 2024-04-16T05:49:17.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 404, success, outsize: 120 2024-04-16T05:49:17.484 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:17 smithi122 ceph-mon[80930]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/72 objects degraded (16.667%), 7 pgs degraded) 2024-04-16T05:49:17.484 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:17 smithi122 ceph-mon[80930]: Cluster is now healthy 2024-04-16T05:49:17.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:17 smithi027 ceph-mon[93635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/72 objects degraded (16.667%), 7 pgs degraded) 2024-04-16T05:49:17.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:17 smithi027 ceph-mon[93635]: Cluster is now healthy 2024-04-16T05:49:18.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:18 smithi122 ceph-mon[80930]: pgmap v141: 65 pgs: 65 active+clean; 584 KiB data, 694 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:18.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:18 smithi027 ceph-mon[93635]: pgmap v141: 65 pgs: 65 active+clean; 584 KiB data, 694 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:19.513 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:19.513 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:19.513 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:49:19.513 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:49:19.513 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:19.513 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:49:19.513 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:49:19.513 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:49:19.513 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:49:19.514 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:19 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-16T05:49:19.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:19.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:19.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:49:19.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:49:19.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:19.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:49:19.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:49:19.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:49:19.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:49:19.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:19 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-16T05:49:20.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:20 smithi122 ceph-mon[80930]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-16T05:49:20.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:20 smithi122 ceph-mon[80930]: Upgrade: osd.4 is safe to restart 2024-04-16T05:49:20.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:20 smithi122 ceph-mon[80930]: pgmap v142: 65 pgs: 65 active+clean; 584 KiB data, 694 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:20.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:20 smithi027 ceph-mon[93635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-16T05:49:20.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:20 smithi027 ceph-mon[93635]: Upgrade: osd.4 is safe to restart 2024-04-16T05:49:20.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:20 smithi027 ceph-mon[93635]: pgmap v142: 65 pgs: 65 active+clean; 584 KiB data, 694 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:22.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:21 smithi122 ceph-mon[80930]: Upgrade: Updating osd.4 2024-04-16T05:49:22.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:21 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:22.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:21 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T05:49:22.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:21 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:49:22.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:21 smithi122 ceph-mon[80930]: Deploying daemon osd.4 on smithi122 2024-04-16T05:49:22.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:21 smithi122 ceph-mon[80930]: pgmap v143: 65 pgs: 65 active+clean; 584 KiB data, 694 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:22.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.184+0000 7fe237400700 1 -- 172.21.15.27:0/1329324226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2300708f0 msgr2=0x7fe230070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:22.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.184+0000 7fe237400700 1 --2- 172.21.15.27:0/1329324226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2300708f0 0x7fe230070d60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe21c004940 tx=0x7fe21c009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:22.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.184+0000 7fe237400700 1 -- 172.21.15.27:0/1329324226 shutdown_connections 2024-04-16T05:49:22.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.184+0000 7fe237400700 1 --2- 172.21.15.27:0/1329324226 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2300708f0 0x7fe230070d60 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:22.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.184+0000 7fe237400700 1 --2- 172.21.15.27:0/1329324226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe230071ea0 0x7fe2300703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:22.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.184+0000 7fe237400700 1 -- 172.21.15.27:0/1329324226 >> 172.21.15.27:0/1329324226 conn(0x7fe2300f8580 msgr2=0x7fe2300fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:22.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.185+0000 7fe237400700 1 -- 172.21.15.27:0/1329324226 shutdown_connections 2024-04-16T05:49:22.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.185+0000 7fe237400700 1 -- 172.21.15.27:0/1329324226 wait complete. 2024-04-16T05:49:22.184 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:21 smithi027 ceph-mon[93635]: Upgrade: Updating osd.4 2024-04-16T05:49:22.184 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:21 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:22.184 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:21 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-16T05:49:22.184 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:21 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:49:22.184 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:21 smithi027 ceph-mon[93635]: Deploying daemon osd.4 on smithi122 2024-04-16T05:49:22.184 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:21 smithi027 ceph-mon[93635]: pgmap v143: 65 pgs: 65 active+clean; 584 KiB data, 694 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:22.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.186+0000 7fe237400700 1 Processor -- start 2024-04-16T05:49:22.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.186+0000 7fe237400700 1 -- start start 2024-04-16T05:49:22.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.186+0000 7fe237400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2300708f0 0x7fe23010fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:22.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.186+0000 7fe237400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe230071ea0 0x7fe2301100e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:22.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.186+0000 7fe237400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2301106e0 con 0x7fe2300708f0 2024-04-16T05:49:22.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.186+0000 7fe237400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe230110850 con 0x7fe230071ea0 2024-04-16T05:49:22.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.187+0000 7fe234a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2300708f0 0x7fe23010fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:22.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.187+0000 7fe234a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2300708f0 0x7fe23010fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50934/0 (socket says 172.21.15.27:50934) 2024-04-16T05:49:22.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.187+0000 7fe234a00700 1 -- 172.21.15.27:0/3386973688 learned_addr learned my addr 172.21.15.27:0/3386973688 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:49:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.187+0000 7fe22fe00700 1 --2- 172.21.15.27:0/3386973688 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe230071ea0 0x7fe2301100e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.187+0000 7fe234a00700 1 -- 172.21.15.27:0/3386973688 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe230071ea0 msgr2=0x7fe2301100e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.187+0000 7fe234a00700 1 --2- 172.21.15.27:0/3386973688 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe230071ea0 0x7fe2301100e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.187+0000 7fe234a00700 1 -- 172.21.15.27:0/3386973688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21c005040 con 0x7fe2300708f0 2024-04-16T05:49:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.188+0000 7fe234a00700 1 --2- 172.21.15.27:0/3386973688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2300708f0 0x7fe23010fba0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe224002a80 tx=0x7fe224004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.188+0000 7fe22cc00700 1 -- 172.21.15.27:0/3386973688 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe224007cc0 con 0x7fe2300708f0 2024-04-16T05:49:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.188+0000 7fe237400700 1 -- 172.21.15.27:0/3386973688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe230115260 con 0x7fe2300708f0 2024-04-16T05:49:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.188+0000 7fe22cc00700 1 -- 172.21.15.27:0/3386973688 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe22400f040 con 0x7fe2300708f0 2024-04-16T05:49:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.188+0000 7fe22cc00700 1 -- 172.21.15.27:0/3386973688 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe22400dad0 con 0x7fe2300708f0 2024-04-16T05:49:22.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.188+0000 7fe237400700 1 -- 172.21.15.27:0/3386973688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe230115820 con 0x7fe2300708f0 2024-04-16T05:49:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.189+0000 7fe237400700 1 -- 172.21.15.27:0/3386973688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe23004c710 con 0x7fe2300708f0 2024-04-16T05:49:22.188 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.191+0000 7fe22cc00700 1 -- 172.21.15.27:0/3386973688 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe22400dc30 con 0x7fe2300708f0 2024-04-16T05:49:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.192+0000 7fe22cc00700 1 --2- 172.21.15.27:0/3386973688 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe21806ec00 0x7fe2180710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:22.189 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.192+0000 7fe22cc00700 1 -- 172.21.15.27:0/3386973688 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6736+0+0 (secure 0 0 0) 0x7fe22408e7a0 con 0x7fe2300708f0 2024-04-16T05:49:22.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.192+0000 7fe22fe00700 1 --2- 172.21.15.27:0/3386973688 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe21806ec00 0x7fe2180710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:22.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.193+0000 7fe22fe00700 1 --2- 172.21.15.27:0/3386973688 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe21806ec00 0x7fe2180710b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe21c00b3b0 tx=0x7fe21c009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:22.195 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.197+0000 7fe22cc00700 1 -- 172.21.15.27:0/3386973688 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe22405af10 con 0x7fe2300708f0 2024-04-16T05:49:22.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.457+0000 7fe237400700 1 -- 172.21.15.27:0/3386973688 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe230103d20 con 0x7fe21806ec00 2024-04-16T05:49:22.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.459+0000 7fe22cc00700 1 -- 172.21.15.27:0/3386973688 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7fe230103d20 con 0x7fe21806ec00 2024-04-16T05:49:22.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.462+0000 7fe237400700 1 -- 172.21.15.27:0/3386973688 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe21806ec00 msgr2=0x7fe2180710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:22.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.462+0000 7fe237400700 1 --2- 172.21.15.27:0/3386973688 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe21806ec00 0x7fe2180710b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe21c00b3b0 tx=0x7fe21c009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:22.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.462+0000 7fe237400700 1 -- 172.21.15.27:0/3386973688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2300708f0 msgr2=0x7fe23010fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:22.460 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.462+0000 7fe237400700 1 --2- 172.21.15.27:0/3386973688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2300708f0 0x7fe23010fba0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe224002a80 tx=0x7fe224004600 comp rx=0 tx=0).stop 2024-04-16T05:49:22.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.463+0000 7fe237400700 1 -- 172.21.15.27:0/3386973688 shutdown_connections 2024-04-16T05:49:22.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.463+0000 7fe237400700 1 --2- 172.21.15.27:0/3386973688 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe21806ec00 0x7fe2180710b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:22.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.463+0000 7fe237400700 1 --2- 172.21.15.27:0/3386973688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2300708f0 0x7fe23010fba0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:22.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.463+0000 7fe237400700 1 --2- 172.21.15.27:0/3386973688 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe230071ea0 0x7fe2301100e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:22.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.463+0000 7fe237400700 1 -- 172.21.15.27:0/3386973688 >> 172.21.15.27:0/3386973688 conn(0x7fe2300f8580 msgr2=0x7fe2300f9ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:22.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.463+0000 7fe237400700 1 -- 172.21.15.27:0/3386973688 shutdown_connections 2024-04-16T05:49:22.461 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.463+0000 7fe237400700 1 -- 172.21.15.27:0/3386973688 wait complete. 2024-04-16T05:49:22.478 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:49:22.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.624+0000 7f5d01e00700 1 -- 172.21.15.27:0/1513086960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cfc103270 msgr2=0x7f5cfc103680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:22.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.624+0000 7f5d01e00700 1 --2- 172.21.15.27:0/1513086960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cfc103270 0x7f5cfc103680 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0004940 tx=0x7f5cf0009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:22.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.624+0000 7f5d01e00700 1 -- 172.21.15.27:0/1513086960 shutdown_connections 2024-04-16T05:49:22.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.624+0000 7f5d01e00700 1 --2- 172.21.15.27:0/1513086960 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cfc104520 0x7f5cfc104990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:22.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.624+0000 7f5d01e00700 1 --2- 172.21.15.27:0/1513086960 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cfc103270 0x7f5cfc103680 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:22.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.624+0000 7f5d01e00700 1 -- 172.21.15.27:0/1513086960 >> 172.21.15.27:0/1513086960 conn(0x7f5cfc072530 msgr2=0x7f5cfc074940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:22.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.624+0000 7f5d01e00700 1 -- 172.21.15.27:0/1513086960 shutdown_connections 2024-04-16T05:49:22.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.624+0000 7f5d01e00700 1 -- 172.21.15.27:0/1513086960 wait complete. 2024-04-16T05:49:22.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.625+0000 7f5d01e00700 1 Processor -- start 2024-04-16T05:49:22.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.625+0000 7f5d01e00700 1 -- start start 2024-04-16T05:49:22.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.626+0000 7f5d01e00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cfc103270 0x7f5cfc113f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:22.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.626+0000 7f5d01e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cfc104520 0x7f5cfc114440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:22.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.626+0000 7f5d01e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc114a40 con 0x7f5cfc104520 2024-04-16T05:49:22.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.626+0000 7f5d01e00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cfc115180 con 0x7f5cfc103270 2024-04-16T05:49:22.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.626+0000 7f5cfbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cfc104520 0x7f5cfc114440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:22.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.626+0000 7f5cfbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cfc104520 0x7f5cfc114440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50946/0 (socket says 172.21.15.27:50946) 2024-04-16T05:49:22.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.626+0000 7f5cfbe00700 1 -- 172.21.15.27:0/2500310995 learned_addr learned my addr 172.21.15.27:0/2500310995 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:49:22.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.626+0000 7f5d00a00700 1 --2- 172.21.15.27:0/2500310995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cfc103270 0x7f5cfc113f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:22.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.627+0000 7f5cfbe00700 1 -- 172.21.15.27:0/2500310995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cfc103270 msgr2=0x7f5cfc113f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:22.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.627+0000 7f5cfbe00700 1 --2- 172.21.15.27:0/2500310995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cfc103270 0x7f5cfc113f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:22.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.627+0000 7f5cfbe00700 1 -- 172.21.15.27:0/2500310995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf0005040 con 0x7f5cfc104520 2024-04-16T05:49:22.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.627+0000 7f5cfbe00700 1 --2- 172.21.15.27:0/2500310995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cfc104520 0x7f5cfc114440 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5ce800c8c0 tx=0x7f5ce80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:22.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.627+0000 7f5cf8c00700 1 -- 172.21.15.27:0/2500310995 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ce800a980 con 0x7f5cfc104520 2024-04-16T05:49:22.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.627+0000 7f5cf8c00700 1 -- 172.21.15.27:0/2500310995 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ce800ce60 con 0x7f5cfc104520 2024-04-16T05:49:22.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.627+0000 7f5d01e00700 1 -- 172.21.15.27:0/2500310995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cfc115460 con 0x7f5cfc104520 2024-04-16T05:49:22.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.627+0000 7f5cf8c00700 1 -- 172.21.15.27:0/2500310995 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ce80115e0 con 0x7f5cfc104520 2024-04-16T05:49:22.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.628+0000 7f5d01e00700 1 -- 172.21.15.27:0/2500310995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cfc115a20 con 0x7f5cfc104520 2024-04-16T05:49:22.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.629+0000 7f5d01e00700 1 -- 172.21.15.27:0/2500310995 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cfc0760a0 con 0x7f5cfc104520 2024-04-16T05:49:22.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.630+0000 7f5cf8c00700 1 -- 172.21.15.27:0/2500310995 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5ce8012400 con 0x7f5cfc104520 2024-04-16T05:49:22.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.631+0000 7f5cf8c00700 1 --2- 172.21.15.27:0/2500310995 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5ce406ec00 0x7f5ce40710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:22.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.631+0000 7f5cf8c00700 1 -- 172.21.15.27:0/2500310995 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f5ce808eae0 con 0x7f5cfc104520 2024-04-16T05:49:22.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.632+0000 7f5d00a00700 1 --2- 172.21.15.27:0/2500310995 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5ce406ec00 0x7f5ce40710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:22.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.632+0000 7f5d00a00700 1 --2- 172.21.15.27:0/2500310995 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5ce406ec00 0x7f5ce40710b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000b3b0 tx=0x7f5cf0009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.636+0000 7f5cf8c00700 1 -- 172.21.15.27:0/2500310995 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5ce805b2d0 con 0x7f5cfc104520 2024-04-16T05:49:22.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.889+0000 7f5d01e00700 1 -- 172.21.15.27:0/2500310995 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5cfc115e10 con 0x7f5ce406ec00 2024-04-16T05:49:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.892+0000 7f5cf8c00700 1 -- 172.21.15.27:0/2500310995 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f5cfc115e10 con 0x7f5ce406ec00 2024-04-16T05:49:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.895+0000 7f5d01e00700 1 -- 172.21.15.27:0/2500310995 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5ce406ec00 msgr2=0x7f5ce40710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.895+0000 7f5d01e00700 1 --2- 172.21.15.27:0/2500310995 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5ce406ec00 0x7f5ce40710b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000b3b0 tx=0x7f5cf0009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.895+0000 7f5d01e00700 1 -- 172.21.15.27:0/2500310995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cfc104520 msgr2=0x7f5cfc114440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.895+0000 7f5d01e00700 1 --2- 172.21.15.27:0/2500310995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cfc104520 0x7f5cfc114440 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5ce800c8c0 tx=0x7f5ce80044c0 comp rx=0 tx=0).stop 2024-04-16T05:49:22.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.895+0000 7f5d01e00700 1 -- 172.21.15.27:0/2500310995 shutdown_connections 2024-04-16T05:49:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.895+0000 7f5d01e00700 1 --2- 172.21.15.27:0/2500310995 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5ce406ec00 0x7f5ce40710b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.895+0000 7f5d01e00700 1 --2- 172.21.15.27:0/2500310995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5cfc103270 0x7f5cfc113f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.895+0000 7f5d01e00700 1 --2- 172.21.15.27:0/2500310995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cfc104520 0x7f5cfc114440 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.895+0000 7f5d01e00700 1 -- 172.21.15.27:0/2500310995 >> 172.21.15.27:0/2500310995 conn(0x7f5cfc072530 msgr2=0x7f5cfc074ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.896+0000 7f5d01e00700 1 -- 172.21.15.27:0/2500310995 shutdown_connections 2024-04-16T05:49:22.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:22.896+0000 7f5d01e00700 1 -- 172.21.15.27:0/2500310995 wait complete. 2024-04-16T05:49:22.912 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:22 smithi122 ceph-mon[80930]: from='client.34338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:49:23.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.052+0000 7f4ff0a00700 1 -- 172.21.15.27:0/3072564844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fec100070 msgr2=0x7f4fec1004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:23.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.052+0000 7f4ff0a00700 1 --2- 172.21.15.27:0/3072564844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fec100070 0x7f4fec1004e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8004940 tx=0x7f4fd8009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:23.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.052+0000 7f4ff0a00700 1 -- 172.21.15.27:0/3072564844 shutdown_connections 2024-04-16T05:49:23.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.052+0000 7f4ff0a00700 1 --2- 172.21.15.27:0/3072564844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fec100070 0x7f4fec1004e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:23.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.052+0000 7f4ff0a00700 1 --2- 172.21.15.27:0/3072564844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fec0fef00 0x7f4fec0ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:23.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.052+0000 7f4ff0a00700 1 -- 172.21.15.27:0/3072564844 >> 172.21.15.27:0/3072564844 conn(0x7f4fec0fa790 msgr2=0x7f4fec0fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:23.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.052+0000 7f4ff0a00700 1 -- 172.21.15.27:0/3072564844 shutdown_connections 2024-04-16T05:49:23.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.052+0000 7f4ff0a00700 1 -- 172.21.15.27:0/3072564844 wait complete. 2024-04-16T05:49:23.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.053+0000 7f4ff0a00700 1 Processor -- start 2024-04-16T05:49:23.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.053+0000 7f4ff0a00700 1 -- start start 2024-04-16T05:49:23.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.054+0000 7f4ff0a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fec0fef00 0x7f4fec10fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:23.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.054+0000 7f4ff0a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fec100070 0x7f4fec10ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:23.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.054+0000 7f4ff0a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec110540 con 0x7f4fec0fef00 2024-04-16T05:49:23.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.054+0000 7f4ff0a00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec110d90 con 0x7f4fec100070 2024-04-16T05:49:23.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.054+0000 7f4feb400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fec0fef00 0x7f4fec10fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:23.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.054+0000 7f4feb400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fec0fef00 0x7f4fec10fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50964/0 (socket says 172.21.15.27:50964) 2024-04-16T05:49:23.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.054+0000 7f4feb400700 1 -- 172.21.15.27:0/1345430183 learned_addr learned my addr 172.21.15.27:0/1345430183 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:49:23.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.054+0000 7f4feaa00700 1 --2- 172.21.15.27:0/1345430183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fec100070 0x7f4fec10ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:23.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.055+0000 7f4feb400700 1 -- 172.21.15.27:0/1345430183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fec100070 msgr2=0x7f4fec10ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:23.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.055+0000 7f4feb400700 1 --2- 172.21.15.27:0/1345430183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fec100070 0x7f4fec10ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:23.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.055+0000 7f4feb400700 1 -- 172.21.15.27:0/1345430183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fd8005040 con 0x7f4fec0fef00 2024-04-16T05:49:23.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.055+0000 7f4feaa00700 1 --2- 172.21.15.27:0/1345430183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fec100070 0x7f4fec10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T05:49:23.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.055+0000 7f4feb400700 1 --2- 172.21.15.27:0/1345430183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fec0fef00 0x7f4fec10fa00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc00eb40 tx=0x7f4fdc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:23.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.055+0000 7f4fe3400700 1 -- 172.21.15.27:0/1345430183 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fdc007cc0 con 0x7f4fec0fef00 2024-04-16T05:49:23.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.055+0000 7f4fe3400700 1 -- 172.21.15.27:0/1345430183 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4fdc00f040 con 0x7f4fec0fef00 2024-04-16T05:49:23.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.055+0000 7f4ff0a00700 1 -- 172.21.15.27:0/1345430183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fec111010 con 0x7f4fec0fef00 2024-04-16T05:49:23.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.055+0000 7f4fe3400700 1 -- 172.21.15.27:0/1345430183 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4fdc009620 con 0x7f4fec0fef00 2024-04-16T05:49:23.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.055+0000 7f4ff0a00700 1 -- 172.21.15.27:0/1345430183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fec111550 con 0x7f4fec0fef00 2024-04-16T05:49:23.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.057+0000 7f4ff0a00700 1 -- 172.21.15.27:0/1345430183 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fec0fe290 con 0x7f4fec0fef00 2024-04-16T05:49:23.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.058+0000 7f4fe3400700 1 -- 172.21.15.27:0/1345430183 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4fdc009780 con 0x7f4fec0fef00 2024-04-16T05:49:23.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.059+0000 7f4fe3400700 1 --2- 172.21.15.27:0/1345430183 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4fd006ec00 0x7f4fd00710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:23.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.059+0000 7f4fe3400700 1 -- 172.21.15.27:0/1345430183 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f4fdc08e610 con 0x7f4fec0fef00 2024-04-16T05:49:23.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.060+0000 7f4feaa00700 1 --2- 172.21.15.27:0/1345430183 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4fd006ec00 0x7f4fd00710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:23.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.060+0000 7f4feaa00700 1 --2- 172.21.15.27:0/1345430183 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4fd006ec00 0x7f4fd00710b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8009fd0 tx=0x7f4fd8009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:23.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.064+0000 7f4fe3400700 1 -- 172.21.15.27:0/1345430183 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4fdc05aeb0 con 0x7f4fec0fef00 2024-04-16T05:49:23.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.304+0000 7f4ff0a00700 1 -- 172.21.15.27:0/1345430183 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f4fec111930 con 0x7f4fd006ec00 2024-04-16T05:49:23.302 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:22 smithi027 ceph-mon[93635]: from='client.34338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:49:23.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.320+0000 7f4fe3400700 1 -- 172.21.15.27:0/1345430183 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f4fec111930 con 0x7f4fd006ec00 2024-04-16T05:49:23.319 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:49:23.319 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (5m) 40s ago 16m 13.0M - 0.25.0 c8568f914cd2 4ed63a8a6b33 2024-04-16T05:49:23.319 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (2m) 40s ago 16m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 4aa1f0006685 2024-04-16T05:49:23.319 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (2m) 12s ago 15m 6656k - 19.0.0-2884-g218a6914 a4e4811815e4 1c1912507d56 2024-04-16T05:49:23.319 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (5m) 40s ago 16m 75.4M - 9.4.7 954c08fa6188 f7ee2a1afd12 2024-04-16T05:49:23.319 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (8m) 40s ago 8m 23.7M - 17.2.7-847-g45317f95 b333581ccdf5 0824e2621be5 2024-04-16T05:49:23.319 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (9m) 40s ago 9m 14.0M - 17.2.7-847-g45317f95 b333581ccdf5 b332a72cc594 2024-04-16T05:49:23.319 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (9m) 12s ago 9m 14.2M - 17.2.7-847-g45317f95 b333581ccdf5 f5e36815ddfa 2024-04-16T05:49:23.319 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (8m) 12s ago 8m 14.1M - 17.2.7-847-g45317f95 b333581ccdf5 755a73c65a7f 2024-04-16T05:49:23.319 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:8443,9283,8765 running (5m) 40s ago 18m 502M - 19.0.0-2884-g218a6914 a4e4811815e4 b69a0b08e12b 2024-04-16T05:49:23.320 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283,8765 running (4m) 12s ago 15m 452M - 19.0.0-2884-g218a6914 a4e4811815e4 32e27bcd7e42 2024-04-16T05:49:23.320 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (4m) 40s ago 18m 45.4M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 299ca337e4ca 2024-04-16T05:49:23.320 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (3m) 12s ago 14m 33.9M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5db5f3037ed0 2024-04-16T05:49:23.320 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (5m) 40s ago 16m 8086k - 1.5.0 0da6a335fe13 5e29e29868b0 2024-04-16T05:49:23.320 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (5m) 12s ago 15m 7482k - 1.5.0 0da6a335fe13 a360922bc621 2024-04-16T05:49:23.320 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (111s) 40s ago 14m 42.5M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 cef03b67bc5a 2024-04-16T05:49:23.320 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (79s) 40s ago 13m 42.2M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3a6ab1fc6e79 2024-04-16T05:49:23.320 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (48s) 40s ago 12m 11.1M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3524731d7277 2024-04-16T05:49:23.320 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (17s) 12s ago 12m 11.1M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 897d3579f929 2024-04-16T05:49:23.320 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (11m) 12s ago 11m 46.7M 2581M 17.2.7-847-g45317f95 b333581ccdf5 7a59908a3e10 2024-04-16T05:49:23.320 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (10m) 12s ago 10m 47.0M 2581M 17.2.7-847-g45317f95 b333581ccdf5 8e4b60163907 2024-04-16T05:49:23.320 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (4m) 40s ago 15m 57.0M - 2.43.0 a07b618ecd1d e70f55ba5ec0 2024-04-16T05:49:23.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.323+0000 7f4ff0a00700 1 -- 172.21.15.27:0/1345430183 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4fd006ec00 msgr2=0x7f4fd00710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:23.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.323+0000 7f4ff0a00700 1 --2- 172.21.15.27:0/1345430183 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4fd006ec00 0x7f4fd00710b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8009fd0 tx=0x7f4fd8009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:23.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.323+0000 7f4ff0a00700 1 -- 172.21.15.27:0/1345430183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fec0fef00 msgr2=0x7f4fec10fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:23.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.323+0000 7f4ff0a00700 1 --2- 172.21.15.27:0/1345430183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fec0fef00 0x7f4fec10fa00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc00eb40 tx=0x7f4fdc00b040 comp rx=0 tx=0).stop 2024-04-16T05:49:23.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.323+0000 7f4ff0a00700 1 -- 172.21.15.27:0/1345430183 shutdown_connections 2024-04-16T05:49:23.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.323+0000 7f4ff0a00700 1 --2- 172.21.15.27:0/1345430183 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4fd006ec00 0x7f4fd00710b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:23.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.323+0000 7f4ff0a00700 1 --2- 172.21.15.27:0/1345430183 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fec0fef00 0x7f4fec10fa00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:23.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.323+0000 7f4ff0a00700 1 --2- 172.21.15.27:0/1345430183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4fec100070 0x7f4fec10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:23.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.323+0000 7f4ff0a00700 1 -- 172.21.15.27:0/1345430183 >> 172.21.15.27:0/1345430183 conn(0x7f4fec0fa790 msgr2=0x7f4fec0fcdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:23.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.324+0000 7f4ff0a00700 1 -- 172.21.15.27:0/1345430183 shutdown_connections 2024-04-16T05:49:23.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.324+0000 7f4ff0a00700 1 -- 172.21.15.27:0/1345430183 wait complete. 2024-04-16T05:49:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.477+0000 7f77b3200700 1 -- 172.21.15.27:0/3310520446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b4104500 msgr2=0x7f77b4104970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.477+0000 7f77b3200700 1 --2- 172.21.15.27:0/3310520446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b4104500 0x7f77b4104970 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f77a0004940 tx=0x7f77a0009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.477+0000 7f77b3200700 1 -- 172.21.15.27:0/3310520446 shutdown_connections 2024-04-16T05:49:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.477+0000 7f77b3200700 1 --2- 172.21.15.27:0/3310520446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b4104500 0x7f77b4104970 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.477+0000 7f77b3200700 1 --2- 172.21.15.27:0/3310520446 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77b4103250 0x7f77b4103660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.477+0000 7f77b3200700 1 -- 172.21.15.27:0/3310520446 >> 172.21.15.27:0/3310520446 conn(0x7f77b4072570 msgr2=0x7f77b40749a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.477+0000 7f77b3200700 1 -- 172.21.15.27:0/3310520446 shutdown_connections 2024-04-16T05:49:23.475 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.477+0000 7f77b3200700 1 -- 172.21.15.27:0/3310520446 wait complete. 2024-04-16T05:49:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.478+0000 7f77b3200700 1 Processor -- start 2024-04-16T05:49:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.478+0000 7f77b3200700 1 -- start start 2024-04-16T05:49:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.478+0000 7f77b3200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b4103250 0x7f77b4113e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.479+0000 7f77b3200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77b4104500 0x7f77b41143b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.479+0000 7f77b3200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b4115120 con 0x7f77b4103250 2024-04-16T05:49:23.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.479+0000 7f77b3200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b4115290 con 0x7f77b4104500 2024-04-16T05:49:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.479+0000 7f77b1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b4103250 0x7f77b4113e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.479+0000 7f77b1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b4103250 0x7f77b4113e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:50984/0 (socket says 172.21.15.27:50984) 2024-04-16T05:49:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.479+0000 7f77b1e00700 1 -- 172.21.15.27:0/3613252216 learned_addr learned my addr 172.21.15.27:0/3613252216 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:49:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.479+0000 7f77b1400700 1 --2- 172.21.15.27:0/3613252216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77b4104500 0x7f77b41143b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:23.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.479+0000 7f77b1e00700 1 -- 172.21.15.27:0/3613252216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77b4104500 msgr2=0x7f77b41143b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.479+0000 7f77b1e00700 1 --2- 172.21.15.27:0/3613252216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77b4104500 0x7f77b41143b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.479+0000 7f77b1e00700 1 -- 172.21.15.27:0/3613252216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a0005040 con 0x7f77b4103250 2024-04-16T05:49:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.480+0000 7f77b1e00700 1 --2- 172.21.15.27:0/3613252216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b4103250 0x7f77b4113e70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f77a400b790 tx=0x7f77a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.480+0000 7f77aa000700 1 -- 172.21.15.27:0/3613252216 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77a400f770 con 0x7f77b4103250 2024-04-16T05:49:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.480+0000 7f77aa000700 1 -- 172.21.15.27:0/3613252216 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f77a400fda0 con 0x7f77b4103250 2024-04-16T05:49:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.480+0000 7f77b3200700 1 -- 172.21.15.27:0/3613252216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b4115510 con 0x7f77b4103250 2024-04-16T05:49:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.480+0000 7f77aa000700 1 -- 172.21.15.27:0/3613252216 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77a400d310 con 0x7f77b4103250 2024-04-16T05:49:23.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.480+0000 7f77b3200700 1 -- 172.21.15.27:0/3613252216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b4115ad0 con 0x7f77b4103250 2024-04-16T05:49:23.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.481+0000 7f77b3200700 1 -- 172.21.15.27:0/3613252216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b40760c0 con 0x7f77b4103250 2024-04-16T05:49:23.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.482+0000 7f77aa000700 1 -- 172.21.15.27:0/3613252216 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f77a4010420 con 0x7f77b4103250 2024-04-16T05:49:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.483+0000 7f77aa000700 1 --2- 172.21.15.27:0/3613252216 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f779806ec00 0x7f77980710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.483+0000 7f77aa000700 1 -- 172.21.15.27:0/3613252216 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f77a408e8a0 con 0x7f77b4103250 2024-04-16T05:49:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.484+0000 7f77b1400700 1 --2- 172.21.15.27:0/3613252216 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f779806ec00 0x7f77980710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.484+0000 7f77b1400700 1 --2- 172.21.15.27:0/3613252216 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f779806ec00 0x7f77980710b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f77a000b3b0 tx=0x7f77a0009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:23.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.488+0000 7f77aa000700 1 -- 172.21.15.27:0/3613252216 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f77a40576a0 con 0x7f77b4103250 2024-04-16T05:49:23.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.808+0000 7f77b3200700 1 -- 172.21.15.27:0/3613252216 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f77b4115da0 con 0x7f77b4103250 2024-04-16T05:49:23.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.810+0000 7f77aa000700 1 -- 172.21.15.27:0/3613252216 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+839 (secure 0 0 0) 0x7f77a405ad60 con 0x7f77b4103250 2024-04-16T05:49:23.808 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:49:23.808 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:49:23.808 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:49:23.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:49:23.808 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:49:23.808 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:49:23.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:49:23.808 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:49:23.808 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2, 2024-04-16T05:49:23.808 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T05:49:23.808 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:49:23.809 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:49:23.809 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T05:49:23.809 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:49:23.809 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:49:23.809 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 6, 2024-04-16T05:49:23.809 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 8 2024-04-16T05:49:23.809 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:49:23.809 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:49:23.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.812+0000 7f77b3200700 1 -- 172.21.15.27:0/3613252216 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f779806ec00 msgr2=0x7f77980710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:23.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.812+0000 7f77b3200700 1 --2- 172.21.15.27:0/3613252216 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f779806ec00 0x7f77980710b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f77a000b3b0 tx=0x7f77a0009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:23.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.812+0000 7f77b3200700 1 -- 172.21.15.27:0/3613252216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b4103250 msgr2=0x7f77b4113e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:23.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.812+0000 7f77b3200700 1 --2- 172.21.15.27:0/3613252216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b4103250 0x7f77b4113e70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f77a400b790 tx=0x7f77a4009250 comp rx=0 tx=0).stop 2024-04-16T05:49:23.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.813+0000 7f77b3200700 1 -- 172.21.15.27:0/3613252216 shutdown_connections 2024-04-16T05:49:23.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.813+0000 7f77b3200700 1 --2- 172.21.15.27:0/3613252216 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f779806ec00 0x7f77980710b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:23.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.813+0000 7f77b3200700 1 --2- 172.21.15.27:0/3613252216 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f77b4103250 0x7f77b4113e70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:23.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.813+0000 7f77b3200700 1 --2- 172.21.15.27:0/3613252216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f77b4104500 0x7f77b41143b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:23.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.813+0000 7f77b3200700 1 -- 172.21.15.27:0/3613252216 >> 172.21.15.27:0/3613252216 conn(0x7f77b4072570 msgr2=0x7f77b4074b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:23.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.813+0000 7f77b3200700 1 -- 172.21.15.27:0/3613252216 shutdown_connections 2024-04-16T05:49:23.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.813+0000 7f77b3200700 1 -- 172.21.15.27:0/3613252216 wait complete. 2024-04-16T05:49:23.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.969+0000 7f9d10e00700 1 -- 172.21.15.27:0/3667276427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d0c0fff70 msgr2=0x7f9d0c102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:23.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.969+0000 7f9d10e00700 1 --2- 172.21.15.27:0/3667276427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d0c0fff70 0x7f9d0c102390 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4004940 tx=0x7f9cf4009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:23.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.969+0000 7f9d10e00700 1 -- 172.21.15.27:0/3667276427 shutdown_connections 2024-04-16T05:49:23.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.969+0000 7f9d10e00700 1 --2- 172.21.15.27:0/3667276427 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d0c0fff70 0x7f9d0c102390 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:23.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.969+0000 7f9d10e00700 1 --2- 172.21.15.27:0/3667276427 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d0c0662f0 0x7f9d0c0ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:23.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.969+0000 7f9d10e00700 1 -- 172.21.15.27:0/3667276427 >> 172.21.15.27:0/3667276427 conn(0x7f9d0c0f74c0 msgr2=0x7f9d0c0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:23.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.969+0000 7f9d10e00700 1 -- 172.21.15.27:0/3667276427 shutdown_connections 2024-04-16T05:49:23.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.969+0000 7f9d10e00700 1 -- 172.21.15.27:0/3667276427 wait complete. 2024-04-16T05:49:23.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.970+0000 7f9d10e00700 1 Processor -- start 2024-04-16T05:49:23.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.970+0000 7f9d10e00700 1 -- start start 2024-04-16T05:49:23.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.970+0000 7f9d10e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d0c0662f0 0x7f9d0c10da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:23.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.970+0000 7f9d10e00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d0c0fff70 0x7f9d0c10dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:23.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.970+0000 7f9d10e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c10e5a0 con 0x7f9d0c0662f0 2024-04-16T05:49:23.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.970+0000 7f9d10e00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c10edf0 con 0x7f9d0c0fff70 2024-04-16T05:49:23.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.971+0000 7f9d09e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d0c0662f0 0x7f9d0c10da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:23.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.971+0000 7f9d09e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d0c0662f0 0x7f9d0c10da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53810/0 (socket says 172.21.15.27:53810) 2024-04-16T05:49:23.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.971+0000 7f9d09e00700 1 -- 172.21.15.27:0/1482568182 learned_addr learned my addr 172.21.15.27:0/1482568182 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:49:23.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.971+0000 7f9d01400700 1 --2- 172.21.15.27:0/1482568182 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d0c0fff70 0x7f9d0c10dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:23.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.971+0000 7f9d09e00700 1 -- 172.21.15.27:0/1482568182 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d0c0fff70 msgr2=0x7f9d0c10dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:23.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.971+0000 7f9d09e00700 1 --2- 172.21.15.27:0/1482568182 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d0c0fff70 0x7f9d0c10dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:23.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.971+0000 7f9d09e00700 1 -- 172.21.15.27:0/1482568182 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf4005040 con 0x7f9d0c0662f0 2024-04-16T05:49:23.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.971+0000 7f9d09e00700 1 --2- 172.21.15.27:0/1482568182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d0c0662f0 0x7f9d0c10da60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc001d00 tx=0x7f9cfc009e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:23.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.972+0000 7f9d02a00700 1 -- 172.21.15.27:0/1482568182 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cfc007d10 con 0x7f9d0c0662f0 2024-04-16T05:49:23.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.972+0000 7f9d10e00700 1 -- 172.21.15.27:0/1482568182 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0c10f0d0 con 0x7f9d0c0662f0 2024-04-16T05:49:23.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.972+0000 7f9d02a00700 1 -- 172.21.15.27:0/1482568182 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9cfc007e70 con 0x7f9d0c0662f0 2024-04-16T05:49:23.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.972+0000 7f9d02a00700 1 -- 172.21.15.27:0/1482568182 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9cfc0145e0 con 0x7f9d0c0662f0 2024-04-16T05:49:23.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.972+0000 7f9d10e00700 1 -- 172.21.15.27:0/1482568182 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d0c10f610 con 0x7f9d0c0662f0 2024-04-16T05:49:23.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.973+0000 7f9d10e00700 1 -- 172.21.15.27:0/1482568182 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d0c05f5f0 con 0x7f9d0c0662f0 2024-04-16T05:49:23.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.974+0000 7f9d02a00700 1 -- 172.21.15.27:0/1482568182 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9cfc00e900 con 0x7f9d0c0662f0 2024-04-16T05:49:23.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.975+0000 7f9d02a00700 1 --2- 172.21.15.27:0/1482568182 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9cec06ec10 0x7f9cec0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:23.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.975+0000 7f9d02a00700 1 -- 172.21.15.27:0/1482568182 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f9cfc08e7d0 con 0x7f9d0c0662f0 2024-04-16T05:49:23.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.975+0000 7f9d01400700 1 --2- 172.21.15.27:0/1482568182 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9cec06ec10 0x7f9cec0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:23.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.976+0000 7f9d01400700 1 --2- 172.21.15.27:0/1482568182 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9cec06ec10 0x7f9cec0710c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4004f90 tx=0x7f9cf4009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:23.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:23.980+0000 7f9d02a00700 1 -- 172.21.15.27:0/1482568182 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9cfc05aff0 con 0x7f9d0c0662f0 2024-04-16T05:49:24.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.259+0000 7f9d10e00700 1 -- 172.21.15.27:0/1482568182 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9d0c10f9e0 con 0x7f9d0c0662f0 2024-04-16T05:49:24.258 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:23 smithi027 ceph-mon[93635]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:49:24.258 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:23 smithi027 ceph-mon[93635]: pgmap v144: 65 pgs: 65 active+clean; 584 KiB data, 694 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:24.258 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:23 smithi027 ceph-mon[93635]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:49:24.258 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:23 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/3613252216' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:49:24.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.260+0000 7f9d02a00700 1 -- 172.21.15.27:0/1482568182 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1880 (secure 0 0 0) 0x7f9cfc05ab80 con 0x7f9d0c0662f0 2024-04-16T05:49:24.259 INFO:teuthology.orchestra.run.smithi027.stdout:e16 2024-04-16T05:49:24.259 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:49:24.259 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:49:24.259 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:49:24.259 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:49:24.259 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:49:24.259 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:49:24.259 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 15 2024-04-16T05:49:24.259 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:49:24.259 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:49:24.259 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:40:32.701649+0000 2024-04-16T05:49:24.260 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:49:24.260 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:49:24.260 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:49:24.260 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:49:24.260 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:49:24.260 INFO:teuthology.orchestra.run.smithi027.stdout:max_xattr_size 65536 2024-04-16T05:49:24.260 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:49:24.260 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:49:24.260 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 46 2024-04-16T05:49:24.260 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:49:24.260 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:49:24.260 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:49:24.260 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=14462} 2024-04-16T05:49:24.260 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:49:24.260 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:49:24.261 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:49:24.261 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:49:24.261 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:49:24.261 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:49:24.261 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:49:24.261 INFO:teuthology.orchestra.run.smithi027.stdout:bal_rank_mask -1 2024-04-16T05:49:24.261 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:49:24.261 INFO:teuthology.orchestra.run.smithi027.stdout:qdb_cluster leader: 0 members: 2024-04-16T05:49:24.261 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:49:24.261 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:49:24.261 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:49:24.261 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:49:24.261 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:49:24.261 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{-1:14474} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2065690722,v1:172.21.15.27:6827/2065690722] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:49:24.262 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:24263} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:49:24.262 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:24273} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:49:24.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.263+0000 7f9d10e00700 1 -- 172.21.15.27:0/1482568182 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9cec06ec10 msgr2=0x7f9cec0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:24.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.263+0000 7f9d10e00700 1 --2- 172.21.15.27:0/1482568182 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9cec06ec10 0x7f9cec0710c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4004f90 tx=0x7f9cf4009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:24.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.263+0000 7f9d10e00700 1 -- 172.21.15.27:0/1482568182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d0c0662f0 msgr2=0x7f9d0c10da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:24.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.263+0000 7f9d10e00700 1 --2- 172.21.15.27:0/1482568182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d0c0662f0 0x7f9d0c10da60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc001d00 tx=0x7f9cfc009e40 comp rx=0 tx=0).stop 2024-04-16T05:49:24.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.263+0000 7f9d10e00700 1 -- 172.21.15.27:0/1482568182 shutdown_connections 2024-04-16T05:49:24.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.263+0000 7f9d10e00700 1 --2- 172.21.15.27:0/1482568182 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9cec06ec10 0x7f9cec0710c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:24.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.263+0000 7f9d10e00700 1 --2- 172.21.15.27:0/1482568182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d0c0662f0 0x7f9d0c10da60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:24.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.263+0000 7f9d10e00700 1 --2- 172.21.15.27:0/1482568182 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9d0c0fff70 0x7f9d0c10dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:24.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.263+0000 7f9d10e00700 1 -- 172.21.15.27:0/1482568182 >> 172.21.15.27:0/1482568182 conn(0x7f9d0c0f74c0 msgr2=0x7f9d0c0f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:24.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.263+0000 7f9d10e00700 1 -- 172.21.15.27:0/1482568182 shutdown_connections 2024-04-16T05:49:24.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.264+0000 7f9d10e00700 1 -- 172.21.15.27:0/1482568182 wait complete. 2024-04-16T05:49:24.263 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2024-04-16T05:49:24.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:23 smithi122 ceph-mon[80930]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:49:24.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:23 smithi122 ceph-mon[80930]: pgmap v144: 65 pgs: 65 active+clean; 584 KiB data, 694 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:24.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:23 smithi122 ceph-mon[80930]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:49:24.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:23 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/3613252216' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:49:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.415+0000 7f6844200700 1 -- 172.21.15.27:0/1194985687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683c0662f0 msgr2=0x7f683c0ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.415+0000 7f6844200700 1 --2- 172.21.15.27:0/1194985687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683c0662f0 0x7f683c0ffa30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6830009ce0 tx=0x7f6830004d10 comp rx=0 tx=0).stop 2024-04-16T05:49:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.415+0000 7f6844200700 1 -- 172.21.15.27:0/1194985687 shutdown_connections 2024-04-16T05:49:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.415+0000 7f6844200700 1 --2- 172.21.15.27:0/1194985687 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f683c0fff70 0x7f683c102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.415+0000 7f6844200700 1 --2- 172.21.15.27:0/1194985687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683c0662f0 0x7f683c0ffa30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.415+0000 7f6844200700 1 -- 172.21.15.27:0/1194985687 >> 172.21.15.27:0/1194985687 conn(0x7f683c0f74c0 msgr2=0x7f683c0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.415+0000 7f6844200700 1 -- 172.21.15.27:0/1194985687 shutdown_connections 2024-04-16T05:49:24.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.415+0000 7f6844200700 1 -- 172.21.15.27:0/1194985687 wait complete. 2024-04-16T05:49:24.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.416+0000 7f6844200700 1 Processor -- start 2024-04-16T05:49:24.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.416+0000 7f6844200700 1 -- start start 2024-04-16T05:49:24.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.416+0000 7f6844200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683c0662f0 0x7f683c10fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:24.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.416+0000 7f6844200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f683c0fff70 0x7f683c110430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:24.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.416+0000 7f6844200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c110a30 con 0x7f683c0662f0 2024-04-16T05:49:24.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.416+0000 7f6844200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c111170 con 0x7f683c0fff70 2024-04-16T05:49:24.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.417+0000 7f6841800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683c0662f0 0x7f683c10fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:24.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.417+0000 7f6841800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683c0662f0 0x7f683c10fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53840/0 (socket says 172.21.15.27:53840) 2024-04-16T05:49:24.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.417+0000 7f6841800700 1 -- 172.21.15.27:0/3291829335 learned_addr learned my addr 172.21.15.27:0/3291829335 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:49:24.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.417+0000 7f6840e00700 1 --2- 172.21.15.27:0/3291829335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f683c0fff70 0x7f683c110430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:24.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.417+0000 7f6841800700 1 -- 172.21.15.27:0/3291829335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f683c0fff70 msgr2=0x7f683c110430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:24.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.417+0000 7f6841800700 1 --2- 172.21.15.27:0/3291829335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f683c0fff70 0x7f683c110430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:24.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.417+0000 7f6841800700 1 -- 172.21.15.27:0/3291829335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6830009940 con 0x7f683c0662f0 2024-04-16T05:49:24.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.417+0000 7f6841800700 1 --2- 172.21.15.27:0/3291829335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683c0662f0 0x7f683c10fef0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6830009910 tx=0x7f6830004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:24.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.418+0000 7f6839600700 1 -- 172.21.15.27:0/3291829335 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f683000e040 con 0x7f683c0662f0 2024-04-16T05:49:24.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.418+0000 7f6839600700 1 -- 172.21.15.27:0/3291829335 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6830008ed0 con 0x7f683c0662f0 2024-04-16T05:49:24.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.418+0000 7f6844200700 1 -- 172.21.15.27:0/3291829335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f683c1113f0 con 0x7f683c0662f0 2024-04-16T05:49:24.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.418+0000 7f6839600700 1 -- 172.21.15.27:0/3291829335 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f683001c800 con 0x7f683c0662f0 2024-04-16T05:49:24.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.418+0000 7f6844200700 1 -- 172.21.15.27:0/3291829335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f683c1119b0 con 0x7f683c0662f0 2024-04-16T05:49:24.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.419+0000 7f6844200700 1 -- 172.21.15.27:0/3291829335 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f683c0f8ff0 con 0x7f683c0662f0 2024-04-16T05:49:24.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.420+0000 7f6839600700 1 -- 172.21.15.27:0/3291829335 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6830012410 con 0x7f683c0662f0 2024-04-16T05:49:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.421+0000 7f6839600700 1 --2- 172.21.15.27:0/3291829335 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f682406ec10 0x7f68240710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.421+0000 7f6839600700 1 -- 172.21.15.27:0/3291829335 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f68300906e0 con 0x7f683c0662f0 2024-04-16T05:49:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.421+0000 7f6840e00700 1 --2- 172.21.15.27:0/3291829335 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f682406ec10 0x7f68240710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:24.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.422+0000 7f6840e00700 1 --2- 172.21.15.27:0/3291829335 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f682406ec10 0x7f68240710c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6828004c40 tx=0x7f6828009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:24.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.426+0000 7f6839600700 1 -- 172.21.15.27:0/3291829335 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f683005ce50 con 0x7f683c0662f0 2024-04-16T05:49:24.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.670+0000 7f6844200700 1 -- 172.21.15.27:0/3291829335 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f683c111c90 con 0x7f682406ec10 2024-04-16T05:49:24.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.674+0000 7f6839600700 1 -- 172.21.15.27:0/3291829335 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f683c111c90 con 0x7f682406ec10 2024-04-16T05:49:24.672 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:49:24.672 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T05:49:24.672 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:49:24.672 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:49:24.672 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [ 2024-04-16T05:49:24.672 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr", 2024-04-16T05:49:24.672 INFO:teuthology.orchestra.run.smithi027.stdout: "mon", 2024-04-16T05:49:24.672 INFO:teuthology.orchestra.run.smithi027.stdout: "crash" 2024-04-16T05:49:24.672 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-16T05:49:24.672 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "10/21 daemons upgraded", 2024-04-16T05:49:24.672 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Currently upgrading osd daemons", 2024-04-16T05:49:24.673 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:49:24.673 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:49:24.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.677+0000 7f6844200700 1 -- 172.21.15.27:0/3291829335 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f682406ec10 msgr2=0x7f68240710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:24.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.677+0000 7f6844200700 1 --2- 172.21.15.27:0/3291829335 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f682406ec10 0x7f68240710c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6828004c40 tx=0x7f6828009040 comp rx=0 tx=0).stop 2024-04-16T05:49:24.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.677+0000 7f6844200700 1 -- 172.21.15.27:0/3291829335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683c0662f0 msgr2=0x7f683c10fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:24.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.677+0000 7f6844200700 1 --2- 172.21.15.27:0/3291829335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683c0662f0 0x7f683c10fef0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6830009910 tx=0x7f6830004d10 comp rx=0 tx=0).stop 2024-04-16T05:49:24.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.677+0000 7f6844200700 1 -- 172.21.15.27:0/3291829335 shutdown_connections 2024-04-16T05:49:24.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.677+0000 7f6844200700 1 --2- 172.21.15.27:0/3291829335 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f682406ec10 0x7f68240710c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:24.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.677+0000 7f6844200700 1 --2- 172.21.15.27:0/3291829335 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f683c0662f0 0x7f683c10fef0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:24.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.677+0000 7f6844200700 1 --2- 172.21.15.27:0/3291829335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f683c0fff70 0x7f683c110430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:24.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.677+0000 7f6844200700 1 -- 172.21.15.27:0/3291829335 >> 172.21.15.27:0/3291829335 conn(0x7f683c0f74c0 msgr2=0x7f683c0fbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:24.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.677+0000 7f6844200700 1 -- 172.21.15.27:0/3291829335 shutdown_connections 2024-04-16T05:49:24.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.677+0000 7f6844200700 1 -- 172.21.15.27:0/3291829335 wait complete. 2024-04-16T05:49:24.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.834+0000 7f8c52c00700 1 -- 172.21.15.27:0/1844809885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c4c103420 msgr2=0x7f8c4c103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:24.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.834+0000 7f8c52c00700 1 --2- 172.21.15.27:0/1844809885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c4c103420 0x7f8c4c103830 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c004940 tx=0x7f8c3c009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:24.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.834+0000 7f8c52c00700 1 -- 172.21.15.27:0/1844809885 shutdown_connections 2024-04-16T05:49:24.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.834+0000 7f8c52c00700 1 --2- 172.21.15.27:0/1844809885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c4c1046d0 0x7f8c4c104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:24.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.834+0000 7f8c52c00700 1 --2- 172.21.15.27:0/1844809885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c4c103420 0x7f8c4c103830 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:24.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.834+0000 7f8c52c00700 1 -- 172.21.15.27:0/1844809885 >> 172.21.15.27:0/1844809885 conn(0x7f8c4c072640 msgr2=0x7f8c4c074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.834+0000 7f8c52c00700 1 -- 172.21.15.27:0/1844809885 shutdown_connections 2024-04-16T05:49:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.834+0000 7f8c52c00700 1 -- 172.21.15.27:0/1844809885 wait complete. 2024-04-16T05:49:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.835+0000 7f8c52c00700 1 Processor -- start 2024-04-16T05:49:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.835+0000 7f8c52c00700 1 -- start start 2024-04-16T05:49:24.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.836+0000 7f8c52c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c4c103420 0x7f8c4c113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.836+0000 7f8c52c00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c4c1046d0 0x7f8c4c1144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.836+0000 7f8c52c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c114aa0 con 0x7f8c4c103420 2024-04-16T05:49:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.836+0000 7f8c52c00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4c114c10 con 0x7f8c4c1046d0 2024-04-16T05:49:24.834 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.836+0000 7f8c4be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c4c103420 0x7f8c4c113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:24.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.836+0000 7f8c4be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c4c103420 0x7f8c4c113f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53860/0 (socket says 172.21.15.27:53860) 2024-04-16T05:49:24.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.836+0000 7f8c4be00700 1 -- 172.21.15.27:0/1198444323 learned_addr learned my addr 172.21.15.27:0/1198444323 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:49:24.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.836+0000 7f8c4b400700 1 --2- 172.21.15.27:0/1198444323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c4c1046d0 0x7f8c4c1144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:24.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.837+0000 7f8c4be00700 1 -- 172.21.15.27:0/1198444323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c4c1046d0 msgr2=0x7f8c4c1144a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:24.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.837+0000 7f8c4be00700 1 --2- 172.21.15.27:0/1198444323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c4c1046d0 0x7f8c4c1144a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:24.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.837+0000 7f8c4be00700 1 -- 172.21.15.27:0/1198444323 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c3c005040 con 0x7f8c4c103420 2024-04-16T05:49:24.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.837+0000 7f8c4b400700 1 --2- 172.21.15.27:0/1198444323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c4c1046d0 0x7f8c4c1144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T05:49:24.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.837+0000 7f8c4be00700 1 --2- 172.21.15.27:0/1198444323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c4c103420 0x7f8c4c113f60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00b3b0 tx=0x7f8c3c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:24.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.837+0000 7f8c43e00700 1 -- 172.21.15.27:0/1198444323 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c3c00e040 con 0x7f8c4c103420 2024-04-16T05:49:24.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.837+0000 7f8c43e00700 1 -- 172.21.15.27:0/1198444323 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c3c004290 con 0x7f8c4c103420 2024-04-16T05:49:24.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.837+0000 7f8c43e00700 1 -- 172.21.15.27:0/1198444323 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c3c01c630 con 0x7f8c4c103420 2024-04-16T05:49:24.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.837+0000 7f8c52c00700 1 -- 172.21.15.27:0/1198444323 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c4c119610 con 0x7f8c4c103420 2024-04-16T05:49:24.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.837+0000 7f8c52c00700 1 -- 172.21.15.27:0/1198444323 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c4c119bd0 con 0x7f8c4c103420 2024-04-16T05:49:24.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.838+0000 7f8c52c00700 1 -- 172.21.15.27:0/1198444323 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c4c04c710 con 0x7f8c4c103420 2024-04-16T05:49:24.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.840+0000 7f8c43e00700 1 -- 172.21.15.27:0/1198444323 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8c3c00bbf0 con 0x7f8c4c103420 2024-04-16T05:49:24.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.841+0000 7f8c43e00700 1 --2- 172.21.15.27:0/1198444323 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8c3006ec00 0x7f8c300710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:24.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.841+0000 7f8c43e00700 1 -- 172.21.15.27:0/1198444323 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f8c3c08fd60 con 0x7f8c4c103420 2024-04-16T05:49:24.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.841+0000 7f8c4b400700 1 --2- 172.21.15.27:0/1198444323 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8c3006ec00 0x7f8c300710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:24.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.842+0000 7f8c4b400700 1 --2- 172.21.15.27:0/1198444323 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8c3006ec00 0x7f8c300710b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8c38009fd0 tx=0x7f8c38009ee0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:24.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:24.846+0000 7f8c43e00700 1 -- 172.21.15.27:0/1198444323 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8c3c094070 con 0x7f8c4c103420 2024-04-16T05:49:25.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:25.187+0000 7f8c52c00700 1 -- 172.21.15.27:0/1198444323 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8c4c10bf10 con 0x7f8c4c103420 2024-04-16T05:49:25.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:25.188+0000 7f8c43e00700 1 -- 172.21.15.27:0/1198444323 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f8c3c05c5d0 con 0x7f8c4c103420 2024-04-16T05:49:25.187 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:49:25.190 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:25.191+0000 7f8c52c00700 1 -- 172.21.15.27:0/1198444323 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8c3006ec00 msgr2=0x7f8c300710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:25.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:25.191+0000 7f8c52c00700 1 --2- 172.21.15.27:0/1198444323 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8c3006ec00 0x7f8c300710b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8c38009fd0 tx=0x7f8c38009ee0 comp rx=0 tx=0).stop 2024-04-16T05:49:25.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:25.191+0000 7f8c52c00700 1 -- 172.21.15.27:0/1198444323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c4c103420 msgr2=0x7f8c4c113f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:25.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:25.191+0000 7f8c52c00700 1 --2- 172.21.15.27:0/1198444323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c4c103420 0x7f8c4c113f60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00b3b0 tx=0x7f8c3c009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:25.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:25.192+0000 7f8c52c00700 1 -- 172.21.15.27:0/1198444323 shutdown_connections 2024-04-16T05:49:25.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:25.192+0000 7f8c52c00700 1 --2- 172.21.15.27:0/1198444323 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8c3006ec00 0x7f8c300710b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:25.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:25.192+0000 7f8c52c00700 1 --2- 172.21.15.27:0/1198444323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8c4c103420 0x7f8c4c113f60 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:25.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:25.192+0000 7f8c52c00700 1 --2- 172.21.15.27:0/1198444323 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8c4c1046d0 0x7f8c4c1144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:25.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:25.192+0000 7f8c52c00700 1 -- 172.21.15.27:0/1198444323 >> 172.21.15.27:0/1198444323 conn(0x7f8c4c072640 msgr2=0x7f8c4c073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:25.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:25.192+0000 7f8c52c00700 1 -- 172.21.15.27:0/1198444323 shutdown_connections 2024-04-16T05:49:25.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:25.192+0000 7f8c52c00700 1 -- 172.21.15.27:0/1198444323 wait complete. 2024-04-16T05:49:25.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:25 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:25.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:25 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1482568182' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:49:25.453 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:25 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:25.453 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:25 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1482568182' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:49:25.708 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:25 smithi122 systemd[1]: Stopping Ceph osd.4 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T05:49:26.021 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:25 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[48013]: 2024-04-16T05:49:25.864+0000 7ff014400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T05:49:26.021 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:25 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[48013]: 2024-04-16T05:49:25.864+0000 7ff014400700 -1 osd.4 73 *** Got signal Terminated *** 2024-04-16T05:49:26.021 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:25 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[48013]: 2024-04-16T05:49:25.864+0000 7ff014400700 -1 osd.4 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T05:49:26.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:26 smithi122 ceph-mon[80930]: from='client.34358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:49:26.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:26 smithi122 ceph-mon[80930]: pgmap v145: 65 pgs: 65 active+clean; 584 KiB data, 694 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:26.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:26 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1198444323' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:49:26.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:26 smithi122 ceph-mon[80930]: osd.4 marked itself down and dead 2024-04-16T05:49:26.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:26 smithi027 ceph-mon[93635]: from='client.34358 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:49:26.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:26 smithi027 ceph-mon[93635]: pgmap v145: 65 pgs: 65 active+clean; 584 KiB data, 694 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:26.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:26 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1198444323' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:49:26.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:26 smithi027 ceph-mon[93635]: osd.4 marked itself down and dead 2024-04-16T05:49:26.887 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:26 smithi122 podman[94687]: 2024-04-16 05:49:26.59741831 +0000 UTC m=+0.858941400 container died 7a59908a3e1086e2d126c23bd87f83f6b958773802bf863b74e1909635253ab7 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-64502-quincy-3-45317f9, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-16T05:49:27.142 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:27 smithi122 podman[94687]: 2024-04-16 05:49:27.00859781 +0000 UTC m=+1.270120900 container cleanup 7a59908a3e1086e2d126c23bd87f83f6b958773802bf863b74e1909635253ab7 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-64502-quincy-3-45317f9, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T05:49:27.142 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:27 smithi122 bash[94687]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4 2024-04-16T05:49:27.521 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:27 smithi122 podman[94705]: 2024-04-16 05:49:27.31825816 +0000 UTC m=+0.700507135 container remove 7a59908a3e1086e2d126c23bd87f83f6b958773802bf863b74e1909635253ab7 (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, ceph=True) 2024-04-16T05:49:27.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:27 smithi122 ceph-mon[80930]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T05:49:27.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:27 smithi122 ceph-mon[80930]: osdmap e74: 6 total, 5 up, 6 in 2024-04-16T05:49:27.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:27 smithi122 ceph-mon[80930]: osdmap e75: 6 total, 5 up, 6 in 2024-04-16T05:49:27.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:27 smithi027 ceph-mon[93635]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T05:49:27.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:27 smithi027 ceph-mon[93635]: osdmap e74: 6 total, 5 up, 6 in 2024-04-16T05:49:27.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:27 smithi027 ceph-mon[93635]: osdmap e75: 6 total, 5 up, 6 in 2024-04-16T05:49:28.437 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:28 smithi027 ceph-mon[93635]: pgmap v147: 65 pgs: 9 peering, 4 stale+active+clean, 52 active+clean; 584 KiB data, 694 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:28.437 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:28 smithi027 ceph-mon[93635]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-16T05:49:28.507 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:28 smithi122 ceph-mon[80930]: pgmap v147: 65 pgs: 9 peering, 4 stale+active+clean, 52 active+clean; 584 KiB data, 694 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:28.507 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:28 smithi122 ceph-mon[80930]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-16T05:49:28.507 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:28 smithi122 podman[94792]: 2024-04-16 05:49:28.061558596 +0000 UTC m=+0.037175999 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T05:49:28.507 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:28 smithi122 podman[94792]: 2024-04-16 05:49:28.185780568 +0000 UTC m=+0.161397997 container create b33f972753b588de417f7d592a0c7a8c9cf521d8cd7b99018d943156478e4ac7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409) 2024-04-16T05:49:28.507 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:28 smithi122 podman[94792]: 2024-04-16 05:49:28.508969205 +0000 UTC m=+0.484586644 container init b33f972753b588de417f7d592a0c7a8c9cf521d8cd7b99018d943156478e4ac7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T05:49:28.772 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:28 smithi122 podman[94792]: 2024-04-16 05:49:28.521073451 +0000 UTC m=+0.496690868 container start b33f972753b588de417f7d592a0c7a8c9cf521d8cd7b99018d943156478e4ac7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-16T05:49:28.772 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:28 smithi122 podman[94792]: 2024-04-16 05:49:28.604928757 +0000 UTC m=+0.580546236 container attach b33f972753b588de417f7d592a0c7a8c9cf521d8cd7b99018d943156478e4ac7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T05:49:29.235 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:28 smithi122 podman[94792]: 2024-04-16 05:49:28.974520673 +0000 UTC m=+0.950138065 container died b33f972753b588de417f7d592a0c7a8c9cf521d8cd7b99018d943156478e4ac7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=) 2024-04-16T05:49:30.006 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:29 smithi122 podman[94818]: 2024-04-16 05:49:29.633486948 +0000 UTC m=+0.638640120 container remove b33f972753b588de417f7d592a0c7a8c9cf521d8cd7b99018d943156478e4ac7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T05:49:30.007 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:29 smithi122 systemd[1]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.4.service: Deactivated successfully. 2024-04-16T05:49:30.007 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:29 smithi122 systemd[1]: Stopped Ceph osd.4 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968. 2024-04-16T05:49:30.007 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:29 smithi122 systemd[1]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.4.service: Consumed 12.697s CPU time. 2024-04-16T05:49:30.257 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:30 smithi122 systemd[1]: Starting Ceph osd.4 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T05:49:30.257 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:30 smithi122 ceph-mon[80930]: pgmap v149: 65 pgs: 1 active+undersized+degraded, 9 peering, 3 stale+active+clean, 6 active+undersized, 46 active+clean; 584 KiB data, 695 MiB used, 536 GiB / 536 GiB avail; 1/72 objects degraded (1.389%) 2024-04-16T05:49:30.257 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:30 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:30.257 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:30 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:49:30.258 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:30 smithi122 ceph-mon[80930]: Health check failed: Degraded data redundancy: 1/72 objects degraded (1.389%), 1 pg degraded (PG_DEGRADED) 2024-04-16T05:49:30.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:30 smithi027 ceph-mon[93635]: pgmap v149: 65 pgs: 1 active+undersized+degraded, 9 peering, 3 stale+active+clean, 6 active+undersized, 46 active+clean; 584 KiB data, 695 MiB used, 536 GiB / 536 GiB avail; 1/72 objects degraded (1.389%) 2024-04-16T05:49:30.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:30 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:30.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:30 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:49:30.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:30 smithi027 ceph-mon[93635]: Health check failed: Degraded data redundancy: 1/72 objects degraded (1.389%), 1 pg degraded (PG_DEGRADED) 2024-04-16T05:49:30.956 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:30 smithi122 podman[94917]: 2024-04-16 05:49:30.511049651 +0000 UTC m=+0.034908147 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T05:49:30.956 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:30 smithi122 podman[94917]: 2024-04-16 05:49:30.651617969 +0000 UTC m=+0.175476489 container create bb03f89dbed79f0526235fcea48ad0285b3c55945fd92a4cb53813a5726a2871 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-16T05:49:30.956 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:30 smithi122 podman[94917]: 2024-04-16 05:49:30.956744133 +0000 UTC m=+0.480602680 container init bb03f89dbed79f0526235fcea48ad0285b3c55945fd92a4cb53813a5726a2871 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409) 2024-04-16T05:49:31.271 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:30 smithi122 podman[94917]: 2024-04-16 05:49:30.969022378 +0000 UTC m=+0.492880905 container start bb03f89dbed79f0526235fcea48ad0285b3c55945fd92a4cb53813a5726a2871 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-activate, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=) 2024-04-16T05:49:31.272 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:31 smithi122 podman[94917]: 2024-04-16 05:49:31.045589079 +0000 UTC m=+0.569447646 container attach bb03f89dbed79f0526235fcea48ad0285b3c55945fd92a4cb53813a5726a2871 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-activate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T05:49:31.714 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 406, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 95038 2024-04-16T05:49:31.715 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 406, success, outsize: 120 2024-04-16T05:49:32.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:32 smithi122 ceph-mon[80930]: pgmap v150: 65 pgs: 2 active+undersized+degraded, 9 peering, 1 stale+active+clean, 13 active+undersized, 40 active+clean; 584 KiB data, 695 MiB used, 536 GiB / 536 GiB avail; 2/72 objects degraded (2.778%) 2024-04-16T05:49:32.273 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:31 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-activate[94931]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-16T05:49:32.273 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:31 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-activate[94931]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-16T05:49:32.273 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:31 smithi122 bash[94917]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-16T05:49:32.273 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:31 smithi122 bash[94917]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-16T05:49:32.274 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:31 smithi122 bash[94917]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-16T05:49:32.274 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:31 smithi122 bash[94917]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-16T05:49:32.274 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:31 smithi122 bash[94917]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-16T05:49:32.274 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:31 smithi122 bash[94917]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-16T05:49:32.274 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:31 smithi122 bash[94917]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-16T05:49:32.274 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:31 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-activate[94931]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-16T05:49:32.274 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:31 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-activate[94931]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-16T05:49:32.275 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:31 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-activate[94931]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2024-04-16T05:49:32.275 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:31 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-activate[94931]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-16T05:49:32.275 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:31 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-activate[94931]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-16T05:49:32.275 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:31 smithi122 podman[94917]: 2024-04-16 05:49:31.907103225 +0000 UTC m=+1.430961738 container died bb03f89dbed79f0526235fcea48ad0285b3c55945fd92a4cb53813a5726a2871 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-16T05:49:32.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:32 smithi027 ceph-mon[93635]: pgmap v150: 65 pgs: 2 active+undersized+degraded, 9 peering, 1 stale+active+clean, 13 active+undersized, 40 active+clean; 584 KiB data, 695 MiB used, 536 GiB / 536 GiB avail; 2/72 objects degraded (2.778%) 2024-04-16T05:49:32.967 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:32 smithi122 podman[95066]: 2024-04-16 05:49:32.623919579 +0000 UTC m=+0.696424987 container remove bb03f89dbed79f0526235fcea48ad0285b3c55945fd92a4cb53813a5726a2871 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-activate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T05:49:33.218 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:33 smithi122 ceph-mon[80930]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-16T05:49:33.218 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:32 smithi122 podman[95130]: 2024-04-16 05:49:32.874238191 +0000 UTC m=+0.032228212 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T05:49:33.218 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:32 smithi122 podman[95130]: 2024-04-16 05:49:32.975114157 +0000 UTC m=+0.133104212 container create fa97758ff7c59ef8fc98c33abe62e6b2993b294271680e5bd274dd053e18642a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T05:49:33.219 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:33 smithi122 podman[95130]: 2024-04-16 05:49:33.220325542 +0000 UTC m=+0.378315651 container init fa97758ff7c59ef8fc98c33abe62e6b2993b294271680e5bd274dd053e18642a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-16T05:49:33.521 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:33 smithi122 podman[95130]: 2024-04-16 05:49:33.231054054 +0000 UTC m=+0.389044094 container start fa97758ff7c59ef8fc98c33abe62e6b2993b294271680e5bd274dd053e18642a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-16T05:49:33.521 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:33 smithi122 ceph-osd[95149]: -- 172.21.15.122:0/1242727786 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556d9903b680 con 0x556d99004c00 2024-04-16T05:49:33.521 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:33 smithi122 bash[95130]: fa97758ff7c59ef8fc98c33abe62e6b2993b294271680e5bd274dd053e18642a 2024-04-16T05:49:33.521 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:33 smithi122 systemd[1]: Started Ceph osd.4 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968. 2024-04-16T05:49:33.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:33 smithi027 ceph-mon[93635]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-16T05:49:34.271 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:33 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T05:49:33.814+0000 7f559fc387c0 -1 Falling back to public interface 2024-04-16T05:49:34.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:34 smithi122 ceph-mon[80930]: pgmap v151: 65 pgs: 8 active+undersized+degraded, 24 active+undersized, 33 active+clean; 584 KiB data, 695 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T05:49:34.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:34 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:34.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:34 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:34.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:34 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:49:34.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:34 smithi122 ceph-mon[80930]: Health check update: Degraded data redundancy: 13/72 objects degraded (18.056%), 8 pgs degraded (PG_DEGRADED) 2024-04-16T05:49:34.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:34 smithi027 ceph-mon[93635]: pgmap v151: 65 pgs: 8 active+undersized+degraded, 24 active+undersized, 33 active+clean; 584 KiB data, 695 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T05:49:34.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:34 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:34.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:34 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:34.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:34 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:49:34.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:34 smithi027 ceph-mon[93635]: Health check update: Degraded data redundancy: 13/72 objects degraded (18.056%), 8 pgs degraded (PG_DEGRADED) 2024-04-16T05:49:36.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:36 smithi122 ceph-mon[80930]: pgmap v152: 65 pgs: 8 active+undersized+degraded, 24 active+undersized, 33 active+clean; 584 KiB data, 695 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T05:49:36.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:36 smithi027 ceph-mon[93635]: pgmap v152: 65 pgs: 8 active+undersized+degraded, 24 active+undersized, 33 active+clean; 584 KiB data, 695 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T05:49:37.771 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:37 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T05:49:37.476+0000 7f559fc387c0 -1 osd.4 73 log_to_monitors true 2024-04-16T05:49:38.771 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 05:49:38 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T05:49:38.467+0000 7f558f000700 -1 osd.4 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T05:49:38.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:38 smithi122 ceph-mon[80930]: pgmap v153: 65 pgs: 8 active+undersized+degraded, 24 active+undersized, 33 active+clean; 584 KiB data, 695 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T05:49:38.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:38 smithi122 ceph-mon[80930]: from='osd.4 [v2:172.21.15.122:6808/2308306674,v1:172.21.15.122:6809/2308306674]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T05:49:38.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:38 smithi122 ceph-mon[80930]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T05:49:38.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:38 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:38.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:38 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:38.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:38 smithi027 ceph-mon[93635]: pgmap v153: 65 pgs: 8 active+undersized+degraded, 24 active+undersized, 33 active+clean; 584 KiB data, 695 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T05:49:38.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:38 smithi027 ceph-mon[93635]: from='osd.4 [v2:172.21.15.122:6808/2308306674,v1:172.21.15.122:6809/2308306674]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T05:49:38.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:38 smithi027 ceph-mon[93635]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-16T05:49:38.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:38 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:38.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:38 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:39.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:39 smithi122 ceph-mon[80930]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-16T05:49:39.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:39 smithi122 ceph-mon[80930]: from='osd.4 [v2:172.21.15.122:6808/2308306674,v1:172.21.15.122:6809/2308306674]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:49:39.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:39 smithi122 ceph-mon[80930]: osdmap e76: 6 total, 5 up, 6 in 2024-04-16T05:49:39.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:39 smithi122 ceph-mon[80930]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:49:39.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:39 smithi027 ceph-mon[93635]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-16T05:49:39.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:39 smithi027 ceph-mon[93635]: from='osd.4 [v2:172.21.15.122:6808/2308306674,v1:172.21.15.122:6809/2308306674]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:49:39.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:39 smithi027 ceph-mon[93635]: osdmap e76: 6 total, 5 up, 6 in 2024-04-16T05:49:39.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:39 smithi027 ceph-mon[93635]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:49:40.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:40 smithi122 ceph-mon[80930]: pgmap v155: 65 pgs: 8 active+undersized+degraded, 24 active+undersized, 33 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T05:49:40.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:40 smithi122 ceph-mon[80930]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T05:49:40.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:40 smithi122 ceph-mon[80930]: osd.4 [v2:172.21.15.122:6808/2308306674,v1:172.21.15.122:6809/2308306674] boot 2024-04-16T05:49:40.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:40 smithi122 ceph-mon[80930]: osdmap e77: 6 total, 6 up, 6 in 2024-04-16T05:49:40.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:40 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T05:49:40.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:40 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:40.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:40 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:40 smithi027 ceph-mon[93635]: pgmap v155: 65 pgs: 8 active+undersized+degraded, 24 active+undersized, 33 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T05:49:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:40 smithi027 ceph-mon[93635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T05:49:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:40 smithi027 ceph-mon[93635]: osd.4 [v2:172.21.15.122:6808/2308306674,v1:172.21.15.122:6809/2308306674] boot 2024-04-16T05:49:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:40 smithi027 ceph-mon[93635]: osdmap e77: 6 total, 6 up, 6 in 2024-04-16T05:49:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-16T05:49:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:41.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:41 smithi122 ceph-mon[80930]: osdmap e78: 6 total, 6 up, 6 in 2024-04-16T05:49:41.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:41 smithi027 ceph-mon[93635]: osdmap e78: 6 total, 6 up, 6 in 2024-04-16T05:49:42.586 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:42 smithi122 ceph-mon[80930]: pgmap v158: 65 pgs: 6 peering, 8 active+undersized+degraded, 18 active+undersized, 33 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T05:49:42.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:42 smithi027 ceph-mon[93635]: pgmap v158: 65 pgs: 6 peering, 8 active+undersized+degraded, 18 active+undersized, 33 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail; 13/72 objects degraded (18.056%) 2024-04-16T05:49:43.726 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 408, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 96870 2024-04-16T05:49:43.726 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 408, success, outsize: 120 2024-04-16T05:49:43.752 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 410, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 96964 2024-04-16T05:49:43.753 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 410, success, outsize: 120 2024-04-16T05:49:43.776 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 412, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97061 2024-04-16T05:49:43.776 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 412, success, outsize: 120 2024-04-16T05:49:43.801 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 414, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97164 2024-04-16T05:49:43.801 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 414, success, outsize: 120 2024-04-16T05:49:43.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:43 smithi027 ceph-mon[93635]: Health check update: Degraded data redundancy: 3/72 objects degraded (4.167%), 3 pgs degraded (PG_DEGRADED) 2024-04-16T05:49:43.829 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 416, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97264 2024-04-16T05:49:43.829 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 416, success, outsize: 120 2024-04-16T05:49:43.855 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 418, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97358 2024-04-16T05:49:43.855 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 418, success, outsize: 120 2024-04-16T05:49:43.883 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 420, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97455 2024-04-16T05:49:43.883 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 420, success, outsize: 120 2024-04-16T05:49:43.912 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 422, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97558 2024-04-16T05:49:43.912 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 422, success, outsize: 120 2024-04-16T05:49:43.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 424, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97658 2024-04-16T05:49:43.942 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 424, success, outsize: 120 2024-04-16T05:49:43.966 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 426, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97752 2024-04-16T05:49:43.966 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 426, success, outsize: 120 2024-04-16T05:49:43.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 428, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97849 2024-04-16T05:49:43.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 428, success, outsize: 120 2024-04-16T05:49:44.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:43 smithi122 ceph-mon[80930]: Health check update: Degraded data redundancy: 3/72 objects degraded (4.167%), 3 pgs degraded (PG_DEGRADED) 2024-04-16T05:49:44.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 430, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 97952 2024-04-16T05:49:44.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 430, success, outsize: 120 2024-04-16T05:49:44.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 432, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98052 2024-04-16T05:49:44.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 432, success, outsize: 120 2024-04-16T05:49:44.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 434, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98146 2024-04-16T05:49:44.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 434, success, outsize: 120 2024-04-16T05:49:44.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 436, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98243 2024-04-16T05:49:44.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 436, success, outsize: 120 2024-04-16T05:49:44.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 438, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98346 2024-04-16T05:49:44.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 438, success, outsize: 120 2024-04-16T05:49:44.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 440, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98446 2024-04-16T05:49:44.155 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 440, success, outsize: 120 2024-04-16T05:49:44.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 442, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98540 2024-04-16T05:49:44.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 442, success, outsize: 120 2024-04-16T05:49:44.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 444, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98637 2024-04-16T05:49:44.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 444, success, outsize: 120 2024-04-16T05:49:44.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 446, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98740 2024-04-16T05:49:44.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 446, success, outsize: 120 2024-04-16T05:49:44.269 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 448, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98840 2024-04-16T05:49:44.270 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 448, success, outsize: 120 2024-04-16T05:49:44.298 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 450, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 98934 2024-04-16T05:49:44.299 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 450, success, outsize: 120 2024-04-16T05:49:44.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 452, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99031 2024-04-16T05:49:44.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 452, success, outsize: 120 2024-04-16T05:49:44.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 454, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99134 2024-04-16T05:49:44.362 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 454, success, outsize: 120 2024-04-16T05:49:44.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 456, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99234 2024-04-16T05:49:44.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 456, success, outsize: 120 2024-04-16T05:49:44.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 458, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99328 2024-04-16T05:49:44.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 458, success, outsize: 120 2024-04-16T05:49:44.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 460, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99425 2024-04-16T05:49:44.455 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 460, success, outsize: 120 2024-04-16T05:49:44.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 462, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 99528 2024-04-16T05:49:44.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 462, success, outsize: 120 2024-04-16T05:49:44.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:44 smithi122 ceph-mon[80930]: pgmap v159: 65 pgs: 6 peering, 3 active+undersized+degraded, 9 active+undersized, 47 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail; 3/72 objects degraded (4.167%) 2024-04-16T05:49:44.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:44 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:44.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:44 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:49:44.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:44 smithi027 ceph-mon[93635]: pgmap v159: 65 pgs: 6 peering, 3 active+undersized+degraded, 9 active+undersized, 47 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail; 3/72 objects degraded (4.167%) 2024-04-16T05:49:44.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:44.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:49:46.719 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:46 smithi122 ceph-mon[80930]: pgmap v160: 65 pgs: 3 peering, 62 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:46.719 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:46 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:46.719 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:46 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:46.719 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:46 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:49:46.719 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:46 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:49:46.719 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:46 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:46.720 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:46 smithi122 ceph-mon[80930]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/72 objects degraded (4.167%), 3 pgs degraded) 2024-04-16T05:49:46.720 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:46 smithi122 ceph-mon[80930]: Cluster is now healthy 2024-04-16T05:49:46.720 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:46 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:49:46.720 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:46 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:49:46.720 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:46 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:49:46.720 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:46 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:49:46.720 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:46 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T05:49:46.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:46 smithi027 ceph-mon[93635]: pgmap v160: 65 pgs: 3 peering, 62 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:46.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:46 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:46.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:46 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:46.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:46 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:49:46.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:46 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:49:46.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:46 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:46.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:46 smithi027 ceph-mon[93635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/72 objects degraded (4.167%), 3 pgs degraded) 2024-04-16T05:49:46.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:46 smithi027 ceph-mon[93635]: Cluster is now healthy 2024-04-16T05:49:46.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:46 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:49:46.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:46 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:49:46.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:46 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:49:46.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:46 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:49:46.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:46 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T05:49:47.746 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:47 smithi122 ceph-mon[80930]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T05:49:47.747 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:47 smithi122 ceph-mon[80930]: Upgrade: osd.5 is safe to restart 2024-04-16T05:49:47.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:47 smithi027 ceph-mon[93635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-16T05:49:47.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:47 smithi027 ceph-mon[93635]: Upgrade: osd.5 is safe to restart 2024-04-16T05:49:48.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:48 smithi122 ceph-mon[80930]: pgmap v161: 65 pgs: 65 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:48.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:48 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:48.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:48 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T05:49:48.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:48 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:49:48.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:48 smithi027 ceph-mon[93635]: pgmap v161: 65 pgs: 65 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:48.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:48 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:48.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:48 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-16T05:49:48.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:48 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:49:49.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:49 smithi122 ceph-mon[80930]: Upgrade: Updating osd.5 2024-04-16T05:49:49.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:49 smithi122 ceph-mon[80930]: Deploying daemon osd.5 on smithi122 2024-04-16T05:49:49.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:49 smithi027 ceph-mon[93635]: Upgrade: Updating osd.5 2024-04-16T05:49:49.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:49 smithi027 ceph-mon[93635]: Deploying daemon osd.5 on smithi122 2024-04-16T05:49:50.721 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:50 smithi122 ceph-mon[80930]: pgmap v162: 65 pgs: 65 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:50.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:50 smithi027 ceph-mon[93635]: pgmap v162: 65 pgs: 65 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:52.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:52 smithi122 ceph-mon[80930]: pgmap v163: 65 pgs: 65 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:52.772 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:52 smithi122 systemd[1]: Stopping Ceph osd.5 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T05:49:52.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:52 smithi027 ceph-mon[93635]: pgmap v163: 65 pgs: 65 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:53.273 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:52 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[54595]: 2024-04-16T05:49:52.944+0000 7f240d800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T05:49:53.273 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:52 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[54595]: 2024-04-16T05:49:52.944+0000 7f240d800700 -1 osd.5 78 *** Got signal Terminated *** 2024-04-16T05:49:53.273 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:52 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[54595]: 2024-04-16T05:49:52.944+0000 7f240d800700 -1 osd.5 78 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T05:49:53.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:53 smithi122 ceph-mon[80930]: osd.5 marked itself down and dead 2024-04-16T05:49:53.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:53 smithi027 ceph-mon[93635]: osd.5 marked itself down and dead 2024-04-16T05:49:54.276 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:54 smithi122 podman[99971]: 2024-04-16 05:49:54.026592462 +0000 UTC m=+1.221299681 container died 8e4b601639075106e0241dfd3b6241c370744fa58e7b4c5118da13cff1c5cbfe (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5, GIT_CLEAN=True, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-64502-quincy-3-45317f9, io.buildah.version=1.33.5) 2024-04-16T05:49:54.569 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:54 smithi122 ceph-mon[80930]: pgmap v164: 65 pgs: 65 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:54.569 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:54 smithi122 ceph-mon[80930]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T05:49:54.569 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:54 smithi122 ceph-mon[80930]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-16T05:49:54.569 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:54 smithi122 ceph-mon[80930]: osdmap e79: 6 total, 5 up, 6 in 2024-04-16T05:49:54.569 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:54 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:54.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:54 smithi027 ceph-mon[93635]: pgmap v164: 65 pgs: 65 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:54.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:54 smithi027 ceph-mon[93635]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T05:49:54.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:54 smithi027 ceph-mon[93635]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-16T05:49:54.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:54 smithi027 ceph-mon[93635]: osdmap e79: 6 total, 5 up, 6 in 2024-04-16T05:49:54.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:54 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:54.961 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:54 smithi122 podman[99971]: 2024-04-16 05:49:54.570843078 +0000 UTC m=+1.765550315 container cleanup 8e4b601639075106e0241dfd3b6241c370744fa58e7b4c5118da13cff1c5cbfe (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, RELEASE=wip-64502-quincy-3-45317f9, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T05:49:54.961 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:54 smithi122 bash[99971]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5 2024-04-16T05:49:54.961 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:54 smithi122 podman[99989]: 2024-04-16 05:49:54.78888763 +0000 UTC m=+0.745925842 container remove 8e4b601639075106e0241dfd3b6241c370744fa58e7b4c5118da13cff1c5cbfe (image=quay.ceph.io/ceph-ci/ceph@sha256:cfbf2a39c6ccd68fccc3510572808da62eeb6de881fd3b3a56659549af4cdb77, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-64502-quincy-3-45317f9, io.buildah.version=1.33.5) 2024-04-16T05:49:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.367+0000 7f1e34800700 1 -- 172.21.15.27:0/1004803982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e2c073170 msgr2=0x7f1e2c073b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.367+0000 7f1e34800700 1 --2- 172.21.15.27:0/1004803982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e2c073170 0x7f1e2c073b80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c004940 tx=0x7f1e1c009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.367+0000 7f1e34800700 1 -- 172.21.15.27:0/1004803982 shutdown_connections 2024-04-16T05:49:55.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.367+0000 7f1e34800700 1 --2- 172.21.15.27:0/1004803982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e2c073170 0x7f1e2c073b80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:55.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.367+0000 7f1e34800700 1 --2- 172.21.15.27:0/1004803982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e2c072820 0x7f1e2c072c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:55.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.367+0000 7f1e34800700 1 -- 172.21.15.27:0/1004803982 >> 172.21.15.27:0/1004803982 conn(0x7f1e2c0fa790 msgr2=0x7f1e2c0fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:55.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.367+0000 7f1e34800700 1 -- 172.21.15.27:0/1004803982 shutdown_connections 2024-04-16T05:49:55.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.367+0000 7f1e34800700 1 -- 172.21.15.27:0/1004803982 wait complete. 2024-04-16T05:49:55.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.368+0000 7f1e34800700 1 Processor -- start 2024-04-16T05:49:55.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.368+0000 7f1e34800700 1 -- start start 2024-04-16T05:49:55.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.369+0000 7f1e34800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e2c072820 0x7f1e2c113f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:55.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.369+0000 7f1e34800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e2c114470 0x7f1e2c1194a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:55.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.369+0000 7f1e34800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c114930 con 0x7f1e2c072820 2024-04-16T05:49:55.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.369+0000 7f1e34800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c114a70 con 0x7f1e2c114470 2024-04-16T05:49:55.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.369+0000 7f1e31e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e2c072820 0x7f1e2c113f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:55.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.369+0000 7f1e31e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e2c072820 0x7f1e2c113f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42370/0 (socket says 172.21.15.27:42370) 2024-04-16T05:49:55.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.369+0000 7f1e31e00700 1 -- 172.21.15.27:0/2208322802 learned_addr learned my addr 172.21.15.27:0/2208322802 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:49:55.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.369+0000 7f1e31400700 1 --2- 172.21.15.27:0/2208322802 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e2c114470 0x7f1e2c1194a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:55.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.369+0000 7f1e31e00700 1 -- 172.21.15.27:0/2208322802 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e2c114470 msgr2=0x7f1e2c1194a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:55.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.369+0000 7f1e31e00700 1 --2- 172.21.15.27:0/2208322802 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e2c114470 0x7f1e2c1194a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:55.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.369+0000 7f1e31e00700 1 -- 172.21.15.27:0/2208322802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1c005040 con 0x7f1e2c072820 2024-04-16T05:49:55.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.370+0000 7f1e31e00700 1 --2- 172.21.15.27:0/2208322802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e2c072820 0x7f1e2c113f30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1e20002a80 tx=0x7f1e20004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:55.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.370+0000 7f1e26000700 1 -- 172.21.15.27:0/2208322802 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e20007cc0 con 0x7f1e2c072820 2024-04-16T05:49:55.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.370+0000 7f1e26000700 1 -- 172.21.15.27:0/2208322802 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e2000f040 con 0x7f1e2c072820 2024-04-16T05:49:55.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.370+0000 7f1e26000700 1 -- 172.21.15.27:0/2208322802 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e2000da30 con 0x7f1e2c072820 2024-04-16T05:49:55.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.370+0000 7f1e34800700 1 -- 172.21.15.27:0/2208322802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2c119a40 con 0x7f1e2c072820 2024-04-16T05:49:55.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.370+0000 7f1e34800700 1 -- 172.21.15.27:0/2208322802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e2c119fb0 con 0x7f1e2c072820 2024-04-16T05:49:55.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.371+0000 7f1e34800700 1 -- 172.21.15.27:0/2208322802 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e2c04c710 con 0x7f1e2c072820 2024-04-16T05:49:55.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.372+0000 7f1e26000700 1 -- 172.21.15.27:0/2208322802 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1e2000db90 con 0x7f1e2c072820 2024-04-16T05:49:55.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.373+0000 7f1e26000700 1 --2- 172.21.15.27:0/2208322802 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1e1406ec00 0x7f1e140710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:55.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.374+0000 7f1e26000700 1 -- 172.21.15.27:0/2208322802 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f1e2008e690 con 0x7f1e2c072820 2024-04-16T05:49:55.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.374+0000 7f1e31400700 1 --2- 172.21.15.27:0/2208322802 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1e1406ec00 0x7f1e140710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:55.372 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.375+0000 7f1e31400700 1 --2- 172.21.15.27:0/2208322802 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1e1406ec00 0x7f1e140710b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c00b3b0 tx=0x7f1e1c009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:55.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.379+0000 7f1e26000700 1 -- 172.21.15.27:0/2208322802 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1e2005ae80 con 0x7f1e2c072820 2024-04-16T05:49:55.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.630+0000 7f1e34800700 1 -- 172.21.15.27:0/2208322802 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e2c10a470 con 0x7f1e1406ec00 2024-04-16T05:49:55.628 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:55 smithi027 ceph-mon[93635]: osdmap e80: 6 total, 5 up, 6 in 2024-04-16T05:49:55.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.633+0000 7f1e26000700 1 -- 172.21.15.27:0/2208322802 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f1e2c10a470 con 0x7f1e1406ec00 2024-04-16T05:49:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.637+0000 7f1e34800700 1 -- 172.21.15.27:0/2208322802 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1e1406ec00 msgr2=0x7f1e140710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.637+0000 7f1e34800700 1 --2- 172.21.15.27:0/2208322802 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1e1406ec00 0x7f1e140710b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c00b3b0 tx=0x7f1e1c009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.637+0000 7f1e34800700 1 -- 172.21.15.27:0/2208322802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e2c072820 msgr2=0x7f1e2c113f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.637+0000 7f1e34800700 1 --2- 172.21.15.27:0/2208322802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e2c072820 0x7f1e2c113f30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1e20002a80 tx=0x7f1e20004600 comp rx=0 tx=0).stop 2024-04-16T05:49:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.637+0000 7f1e34800700 1 -- 172.21.15.27:0/2208322802 shutdown_connections 2024-04-16T05:49:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.637+0000 7f1e34800700 1 --2- 172.21.15.27:0/2208322802 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1e1406ec00 0x7f1e140710b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.637+0000 7f1e34800700 1 --2- 172.21.15.27:0/2208322802 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1e2c072820 0x7f1e2c113f30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.637+0000 7f1e34800700 1 --2- 172.21.15.27:0/2208322802 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1e2c114470 0x7f1e2c1194a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.637+0000 7f1e34800700 1 -- 172.21.15.27:0/2208322802 >> 172.21.15.27:0/2208322802 conn(0x7f1e2c0fa790 msgr2=0x7f1e2c0fbe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.637+0000 7f1e34800700 1 -- 172.21.15.27:0/2208322802 shutdown_connections 2024-04-16T05:49:55.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.637+0000 7f1e34800700 1 -- 172.21.15.27:0/2208322802 wait complete. 2024-04-16T05:49:55.651 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:49:55.767 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:55 smithi122 ceph-mon[80930]: osdmap e80: 6 total, 5 up, 6 in 2024-04-16T05:49:55.768 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:55 smithi122 podman[100085]: 2024-04-16 05:49:55.525940938 +0000 UTC m=+0.037226893 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T05:49:55.768 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:55 smithi122 podman[100085]: 2024-04-16 05:49:55.623157058 +0000 UTC m=+0.134443043 container create 2123096d3d05b225280c8c7c5919d43b7244c00f7cb3d23b48255883e3b86eb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-16T05:49:55.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.802+0000 7f15eec00700 1 -- 172.21.15.27:0/430153266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e80fe070 msgr2=0x7f15e80fe4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:55.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.802+0000 7f15eec00700 1 --2- 172.21.15.27:0/430153266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e80fe070 0x7f15e80fe4e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f15d4004940 tx=0x7f15d4009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:55.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.802+0000 7f15eec00700 1 -- 172.21.15.27:0/430153266 shutdown_connections 2024-04-16T05:49:55.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.802+0000 7f15eec00700 1 --2- 172.21.15.27:0/430153266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e80fe070 0x7f15e80fe4e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:55.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.802+0000 7f15eec00700 1 --2- 172.21.15.27:0/430153266 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15e80fcf00 0x7f15e80fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:55.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.802+0000 7f15eec00700 1 -- 172.21.15.27:0/430153266 >> 172.21.15.27:0/430153266 conn(0x7f15e80f85e0 msgr2=0x7f15e80fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:55.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.802+0000 7f15eec00700 1 -- 172.21.15.27:0/430153266 shutdown_connections 2024-04-16T05:49:55.800 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.802+0000 7f15eec00700 1 -- 172.21.15.27:0/430153266 wait complete. 2024-04-16T05:49:55.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.803+0000 7f15eec00700 1 Processor -- start 2024-04-16T05:49:55.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.803+0000 7f15eec00700 1 -- start start 2024-04-16T05:49:55.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.803+0000 7f15eec00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15e80fcf00 0x7f15e810d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:55.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.804+0000 7f15eec00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e80fe070 0x7f15e810df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:55.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.804+0000 7f15eec00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e810e500 con 0x7f15e80fe070 2024-04-16T05:49:55.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.804+0000 7f15eec00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e810ed50 con 0x7f15e80fcf00 2024-04-16T05:49:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.804+0000 7f15e7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e80fe070 0x7f15e810df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.804+0000 7f15e7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e80fe070 0x7f15e810df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42386/0 (socket says 172.21.15.27:42386) 2024-04-16T05:49:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.804+0000 7f15e7400700 1 -- 172.21.15.27:0/3469354111 learned_addr learned my addr 172.21.15.27:0/3469354111 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:49:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.804+0000 7f15e7e00700 1 --2- 172.21.15.27:0/3469354111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15e80fcf00 0x7f15e810d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.804+0000 7f15e7400700 1 -- 172.21.15.27:0/3469354111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15e80fcf00 msgr2=0x7f15e810d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.804+0000 7f15e7400700 1 --2- 172.21.15.27:0/3469354111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15e80fcf00 0x7f15e810d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.804+0000 7f15e7400700 1 -- 172.21.15.27:0/3469354111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d4005040 con 0x7f15e80fe070 2024-04-16T05:49:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.805+0000 7f15e7400700 1 --2- 172.21.15.27:0/3469354111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e80fe070 0x7f15e810df00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f15d400b0d0 tx=0x7f15d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.805+0000 7f15dfe00700 1 -- 172.21.15.27:0/3469354111 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15d400e070 con 0x7f15e80fe070 2024-04-16T05:49:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.805+0000 7f15dfe00700 1 -- 172.21.15.27:0/3469354111 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15d400b410 con 0x7f15e80fe070 2024-04-16T05:49:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.805+0000 7f15dfe00700 1 -- 172.21.15.27:0/3469354111 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15d40125e0 con 0x7f15e80fe070 2024-04-16T05:49:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.805+0000 7f15eec00700 1 -- 172.21.15.27:0/3469354111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e810efd0 con 0x7f15e80fe070 2024-04-16T05:49:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.805+0000 7f15eec00700 1 -- 172.21.15.27:0/3469354111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e810f4c0 con 0x7f15e80fe070 2024-04-16T05:49:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.806+0000 7f15eec00700 1 -- 172.21.15.27:0/3469354111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e805f5f0 con 0x7f15e80fe070 2024-04-16T05:49:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.808+0000 7f15dfe00700 1 -- 172.21.15.27:0/3469354111 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f15d400b580 con 0x7f15e80fe070 2024-04-16T05:49:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.809+0000 7f15dfe00700 1 --2- 172.21.15.27:0/3469354111 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f15cc06ec10 0x7f15cc0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.809+0000 7f15dfe00700 1 -- 172.21.15.27:0/3469354111 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f15d408fae0 con 0x7f15e80fe070 2024-04-16T05:49:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.810+0000 7f15e7e00700 1 --2- 172.21.15.27:0/3469354111 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f15cc06ec10 0x7f15cc0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:55.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.810+0000 7f15e7e00700 1 --2- 172.21.15.27:0/3469354111 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f15cc06ec10 0x7f15cc0710c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f15d8001470 tx=0x7f15d800b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:55.811 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:55.813+0000 7f15dfe00700 1 -- 172.21.15.27:0/3469354111 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f15d405c350 con 0x7f15e80fe070 2024-04-16T05:49:56.022 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:55 smithi122 podman[100085]: 2024-04-16 05:49:55.851744628 +0000 UTC m=+0.363030620 container init 2123096d3d05b225280c8c7c5919d43b7244c00f7cb3d23b48255883e3b86eb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T05:49:56.022 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:55 smithi122 podman[100085]: 2024-04-16 05:49:55.861445712 +0000 UTC m=+0.372731666 container start 2123096d3d05b225280c8c7c5919d43b7244c00f7cb3d23b48255883e3b86eb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-16T05:49:56.022 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:55 smithi122 podman[100085]: 2024-04-16 05:49:55.967413226 +0000 UTC m=+0.478699224 container attach 2123096d3d05b225280c8c7c5919d43b7244c00f7cb3d23b48255883e3b86eb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-16T05:49:56.056 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.058+0000 7f15eec00700 1 -- 172.21.15.27:0/3469354111 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f15e810f8b0 con 0x7f15cc06ec10 2024-04-16T05:49:56.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.060+0000 7f15dfe00700 1 -- 172.21.15.27:0/3469354111 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f15e810f8b0 con 0x7f15cc06ec10 2024-04-16T05:49:56.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.062+0000 7f15eec00700 1 -- 172.21.15.27:0/3469354111 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f15cc06ec10 msgr2=0x7f15cc0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:56.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.062+0000 7f15eec00700 1 --2- 172.21.15.27:0/3469354111 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f15cc06ec10 0x7f15cc0710c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f15d8001470 tx=0x7f15d800b040 comp rx=0 tx=0).stop 2024-04-16T05:49:56.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.063+0000 7f15eec00700 1 -- 172.21.15.27:0/3469354111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e80fe070 msgr2=0x7f15e810df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:56.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.063+0000 7f15eec00700 1 --2- 172.21.15.27:0/3469354111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e80fe070 0x7f15e810df00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f15d400b0d0 tx=0x7f15d4009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:56.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.063+0000 7f15eec00700 1 -- 172.21.15.27:0/3469354111 shutdown_connections 2024-04-16T05:49:56.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.063+0000 7f15eec00700 1 --2- 172.21.15.27:0/3469354111 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f15cc06ec10 0x7f15cc0710c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:56.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.063+0000 7f15eec00700 1 --2- 172.21.15.27:0/3469354111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15e80fcf00 0x7f15e810d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:56.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.063+0000 7f15eec00700 1 --2- 172.21.15.27:0/3469354111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f15e80fe070 0x7f15e810df00 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:56.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.063+0000 7f15eec00700 1 -- 172.21.15.27:0/3469354111 >> 172.21.15.27:0/3469354111 conn(0x7f15e80f85e0 msgr2=0x7f15e80f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:56.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.063+0000 7f15eec00700 1 -- 172.21.15.27:0/3469354111 shutdown_connections 2024-04-16T05:49:56.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.063+0000 7f15eec00700 1 -- 172.21.15.27:0/3469354111 wait complete. 2024-04-16T05:49:56.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.221+0000 7f915fc00700 1 -- 172.21.15.27:0/1924612578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9158104650 msgr2=0x7f9158104ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.221+0000 7f915fc00700 1 --2- 172.21.15.27:0/1924612578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9158104650 0x7f9158104ac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9144004940 tx=0x7f9144009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.221+0000 7f915fc00700 1 -- 172.21.15.27:0/1924612578 shutdown_connections 2024-04-16T05:49:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.221+0000 7f915fc00700 1 --2- 172.21.15.27:0/1924612578 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9158104650 0x7f9158104ac0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.222+0000 7f915fc00700 1 --2- 172.21.15.27:0/1924612578 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91581033f0 0x7f9158103800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.222+0000 7f915fc00700 1 -- 172.21.15.27:0/1924612578 >> 172.21.15.27:0/1924612578 conn(0x7f9158072530 msgr2=0x7f9158074960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.222+0000 7f915fc00700 1 -- 172.21.15.27:0/1924612578 shutdown_connections 2024-04-16T05:49:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.222+0000 7f915fc00700 1 -- 172.21.15.27:0/1924612578 wait complete. 2024-04-16T05:49:56.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.223+0000 7f915fc00700 1 Processor -- start 2024-04-16T05:49:56.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.223+0000 7f915fc00700 1 -- start start 2024-04-16T05:49:56.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.223+0000 7f915fc00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91581033f0 0x7f9158113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:56.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.224+0000 7f915fc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9158104650 0x7f9158114510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:56.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.224+0000 7f915fc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9158114b10 con 0x7f9158104650 2024-04-16T05:49:56.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.224+0000 7f915fc00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9158115360 con 0x7f91581033f0 2024-04-16T05:49:56.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.224+0000 7f9157e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9158104650 0x7f9158114510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:56.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.224+0000 7f9157e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9158104650 0x7f9158114510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42414/0 (socket says 172.21.15.27:42414) 2024-04-16T05:49:56.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.224+0000 7f9157e00700 1 -- 172.21.15.27:0/3364401623 learned_addr learned my addr 172.21.15.27:0/3364401623 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:49:56.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.224+0000 7f915d200700 1 --2- 172.21.15.27:0/3364401623 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91581033f0 0x7f9158113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:56.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.224+0000 7f9157e00700 1 -- 172.21.15.27:0/3364401623 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91581033f0 msgr2=0x7f9158113fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:56.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.224+0000 7f9157e00700 1 --2- 172.21.15.27:0/3364401623 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91581033f0 0x7f9158113fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:56.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.224+0000 7f9157e00700 1 -- 172.21.15.27:0/3364401623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9144005040 con 0x7f9158104650 2024-04-16T05:49:56.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.225+0000 7f915d200700 1 --2- 172.21.15.27:0/3364401623 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91581033f0 0x7f9158113fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T05:49:56.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.225+0000 7f9157e00700 1 --2- 172.21.15.27:0/3364401623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9158104650 0x7f9158114510 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f914400b3b0 tx=0x7f9144009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:56.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.225+0000 7f9154c00700 1 -- 172.21.15.27:0/3364401623 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f914400e040 con 0x7f9158104650 2024-04-16T05:49:56.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.225+0000 7f9154c00700 1 -- 172.21.15.27:0/3364401623 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9144004290 con 0x7f9158104650 2024-04-16T05:49:56.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.225+0000 7f9154c00700 1 -- 172.21.15.27:0/3364401623 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f914401c630 con 0x7f9158104650 2024-04-16T05:49:56.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.225+0000 7f915fc00700 1 -- 172.21.15.27:0/3364401623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91581155e0 con 0x7f9158104650 2024-04-16T05:49:56.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.226+0000 7f915fc00700 1 -- 172.21.15.27:0/3364401623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9158115b20 con 0x7f9158104650 2024-04-16T05:49:56.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.227+0000 7f915fc00700 1 -- 172.21.15.27:0/3364401623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91580761d0 con 0x7f9158104650 2024-04-16T05:49:56.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.228+0000 7f9154c00700 1 -- 172.21.15.27:0/3364401623 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f914400bbf0 con 0x7f9158104650 2024-04-16T05:49:56.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.229+0000 7f9154c00700 1 --2- 172.21.15.27:0/3364401623 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f914006ec00 0x7f91400710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:56.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.229+0000 7f9154c00700 1 -- 172.21.15.27:0/3364401623 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f914408fd60 con 0x7f9158104650 2024-04-16T05:49:56.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.229+0000 7f915d200700 1 --2- 172.21.15.27:0/3364401623 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f914006ec00 0x7f91400710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:56.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.230+0000 7f915d200700 1 --2- 172.21.15.27:0/3364401623 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f914006ec00 0x7f91400710b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f914c009fd0 tx=0x7f914c009ee0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:56.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.234+0000 7f9154c00700 1 -- 172.21.15.27:0/3364401623 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9144094070 con 0x7f9158104650 2024-04-16T05:49:56.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.475+0000 7f915fc00700 1 -- 172.21.15.27:0/3364401623 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9158115f10 con 0x7f914006ec00 2024-04-16T05:49:56.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.491+0000 7f9154c00700 1 -- 172.21.15.27:0/3364401623 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f9158115f10 con 0x7f914006ec00 2024-04-16T05:49:56.490 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:49:56.490 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (6m) 73s ago 16m 13.0M - 0.25.0 c8568f914cd2 4ed63a8a6b33 2024-04-16T05:49:56.490 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (3m) 73s ago 17m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 4aa1f0006685 2024-04-16T05:49:56.490 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (2m) 18s ago 15m 6656k - 19.0.0-2884-g218a6914 a4e4811815e4 1c1912507d56 2024-04-16T05:49:56.490 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (6m) 73s ago 16m 75.4M - 9.4.7 954c08fa6188 f7ee2a1afd12 2024-04-16T05:49:56.490 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (9m) 73s ago 9m 23.7M - 17.2.7-847-g45317f95 b333581ccdf5 0824e2621be5 2024-04-16T05:49:56.490 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (9m) 73s ago 9m 14.0M - 17.2.7-847-g45317f95 b333581ccdf5 b332a72cc594 2024-04-16T05:49:56.490 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (9m) 18s ago 9m 14.3M - 17.2.7-847-g45317f95 b333581ccdf5 f5e36815ddfa 2024-04-16T05:49:56.490 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (9m) 18s ago 9m 14.2M - 17.2.7-847-g45317f95 b333581ccdf5 755a73c65a7f 2024-04-16T05:49:56.490 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:8443,9283,8765 running (5m) 73s ago 18m 502M - 19.0.0-2884-g218a6914 a4e4811815e4 b69a0b08e12b 2024-04-16T05:49:56.490 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283,8765 running (4m) 18s ago 15m 453M - 19.0.0-2884-g218a6914 a4e4811815e4 32e27bcd7e42 2024-04-16T05:49:56.490 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (4m) 73s ago 18m 45.4M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 299ca337e4ca 2024-04-16T05:49:56.490 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (4m) 18s ago 15m 34.4M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5db5f3037ed0 2024-04-16T05:49:56.490 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (6m) 73s ago 17m 8086k - 1.5.0 0da6a335fe13 5e29e29868b0 2024-04-16T05:49:56.491 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (5m) 18s ago 15m 7922k - 1.5.0 0da6a335fe13 a360922bc621 2024-04-16T05:49:56.491 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (2m) 73s ago 14m 42.5M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 cef03b67bc5a 2024-04-16T05:49:56.491 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (112s) 73s ago 13m 42.2M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3a6ab1fc6e79 2024-04-16T05:49:56.491 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (81s) 73s ago 13m 11.1M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3524731d7277 2024-04-16T05:49:56.491 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (50s) 18s ago 12m 42.4M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 897d3579f929 2024-04-16T05:49:56.491 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (23s) 18s ago 12m 11.2M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 fa97758ff7c5 2024-04-16T05:49:56.491 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (11m) 18s ago 11m 47.1M 2581M 17.2.7-847-g45317f95 b333581ccdf5 8e4b60163907 2024-04-16T05:49:56.491 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (5m) 73s ago 16m 57.0M - 2.43.0 a07b618ecd1d e70f55ba5ec0 2024-04-16T05:49:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.495+0000 7f915fc00700 1 -- 172.21.15.27:0/3364401623 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f914006ec00 msgr2=0x7f91400710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.495+0000 7f915fc00700 1 --2- 172.21.15.27:0/3364401623 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f914006ec00 0x7f91400710b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f914c009fd0 tx=0x7f914c009ee0 comp rx=0 tx=0).stop 2024-04-16T05:49:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.495+0000 7f915fc00700 1 -- 172.21.15.27:0/3364401623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9158104650 msgr2=0x7f9158114510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.495+0000 7f915fc00700 1 --2- 172.21.15.27:0/3364401623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9158104650 0x7f9158114510 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f914400b3b0 tx=0x7f9144009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.495+0000 7f915fc00700 1 -- 172.21.15.27:0/3364401623 shutdown_connections 2024-04-16T05:49:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.495+0000 7f915fc00700 1 --2- 172.21.15.27:0/3364401623 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f914006ec00 0x7f91400710b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.495+0000 7f915fc00700 1 --2- 172.21.15.27:0/3364401623 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91581033f0 0x7f9158113fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:56.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.495+0000 7f915fc00700 1 --2- 172.21.15.27:0/3364401623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9158104650 0x7f9158114510 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.495+0000 7f915fc00700 1 -- 172.21.15.27:0/3364401623 >> 172.21.15.27:0/3364401623 conn(0x7f9158072530 msgr2=0x7f9158073c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.495+0000 7f915fc00700 1 -- 172.21.15.27:0/3364401623 shutdown_connections 2024-04-16T05:49:56.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.496+0000 7f915fc00700 1 -- 172.21.15.27:0/3364401623 wait complete. 2024-04-16T05:49:56.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.653+0000 7f506c200700 1 -- 172.21.15.27:0/3980996844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50641002d0 msgr2=0x7f5064100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:56.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.653+0000 7f506c200700 1 --2- 172.21.15.27:0/3980996844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50641002d0 0x7f5064100740 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5050004940 tx=0x7f5050009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:56.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.653+0000 7f506c200700 1 -- 172.21.15.27:0/3980996844 shutdown_connections 2024-04-16T05:49:56.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.653+0000 7f506c200700 1 --2- 172.21.15.27:0/3980996844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50641002d0 0x7f5064100740 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:56.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.653+0000 7f506c200700 1 --2- 172.21.15.27:0/3980996844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50640ff020 0x7f50640ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:56.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.653+0000 7f506c200700 1 -- 172.21.15.27:0/3980996844 >> 172.21.15.27:0/3980996844 conn(0x7f50640fa950 msgr2=0x7f50640fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:56.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.653+0000 7f506c200700 1 -- 172.21.15.27:0/3980996844 shutdown_connections 2024-04-16T05:49:56.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.653+0000 7f506c200700 1 -- 172.21.15.27:0/3980996844 wait complete. 2024-04-16T05:49:56.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.654+0000 7f506c200700 1 Processor -- start 2024-04-16T05:49:56.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.654+0000 7f506c200700 1 -- start start 2024-04-16T05:49:56.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.655+0000 7f506c200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50640ff020 0x7f506410fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:56.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.655+0000 7f506c200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50641002d0 0x7f50641100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:56.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.655+0000 7f506c200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50641106b0 con 0x7f50640ff020 2024-04-16T05:49:56.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.655+0000 7f506c200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5064110820 con 0x7f50641002d0 2024-04-16T05:49:56.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.655+0000 7f5069800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50640ff020 0x7f506410fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:56.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.655+0000 7f5069800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50640ff020 0x7f506410fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42434/0 (socket says 172.21.15.27:42434) 2024-04-16T05:49:56.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.655+0000 7f5069800700 1 -- 172.21.15.27:0/3421747723 learned_addr learned my addr 172.21.15.27:0/3421747723 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:49:56.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.655+0000 7f5068e00700 1 --2- 172.21.15.27:0/3421747723 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50641002d0 0x7f50641100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:56.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.655+0000 7f5069800700 1 -- 172.21.15.27:0/3421747723 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50641002d0 msgr2=0x7f50641100b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:56.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.656+0000 7f5069800700 1 --2- 172.21.15.27:0/3421747723 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50641002d0 0x7f50641100b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:56.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.656+0000 7f5069800700 1 -- 172.21.15.27:0/3421747723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5050005040 con 0x7f50640ff020 2024-04-16T05:49:56.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.656+0000 7f5069800700 1 --2- 172.21.15.27:0/3421747723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50640ff020 0x7f506410fb70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5058002a50 tx=0x7f50580046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:56.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.656+0000 7f5061600700 1 -- 172.21.15.27:0/3421747723 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5058007cc0 con 0x7f50640ff020 2024-04-16T05:49:56.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.656+0000 7f5061600700 1 -- 172.21.15.27:0/3421747723 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f505800f040 con 0x7f50640ff020 2024-04-16T05:49:56.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.656+0000 7f506c200700 1 -- 172.21.15.27:0/3421747723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5064115280 con 0x7f50640ff020 2024-04-16T05:49:56.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.656+0000 7f5061600700 1 -- 172.21.15.27:0/3421747723 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5058009640 con 0x7f50640ff020 2024-04-16T05:49:56.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.656+0000 7f506c200700 1 -- 172.21.15.27:0/3421747723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5064115840 con 0x7f50640ff020 2024-04-16T05:49:56.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.657+0000 7f506c200700 1 -- 172.21.15.27:0/3421747723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f506404c710 con 0x7f50640ff020 2024-04-16T05:49:56.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.658+0000 7f5061600700 1 -- 172.21.15.27:0/3421747723 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f50580097a0 con 0x7f50640ff020 2024-04-16T05:49:56.656 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:56 smithi122 ceph-mon[80930]: pgmap v167: 65 pgs: 9 peering, 9 stale+active+clean, 47 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:56.656 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:56 smithi122 ceph-mon[80930]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-16T05:49:56.657 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:56 smithi122 podman[100085]: 2024-04-16 05:49:56.29353937 +0000 UTC m=+0.804825372 container died 2123096d3d05b225280c8c7c5919d43b7244c00f7cb3d23b48255883e3b86eb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T05:49:56.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.659+0000 7f5061600700 1 --2- 172.21.15.27:0/3421747723 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f504c06ec00 0x7f504c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:56.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.660+0000 7f5061600700 1 -- 172.21.15.27:0/3421747723 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f505808e720 con 0x7f50640ff020 2024-04-16T05:49:56.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.660+0000 7f5068e00700 1 --2- 172.21.15.27:0/3421747723 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f504c06ec00 0x7f504c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:56.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.661+0000 7f5068e00700 1 --2- 172.21.15.27:0/3421747723 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f504c06ec00 0x7f504c0710b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5050000c00 tx=0x7f5050009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:56.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.664+0000 7f5061600700 1 -- 172.21.15.27:0/3421747723 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f505805af10 con 0x7f50640ff020 2024-04-16T05:49:56.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:56 smithi027 ceph-mon[93635]: pgmap v167: 65 pgs: 9 peering, 9 stale+active+clean, 47 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:49:56.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:56 smithi027 ceph-mon[93635]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-16T05:49:56.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.974+0000 7f506c200700 1 -- 172.21.15.27:0/3421747723 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5064107b10 con 0x7f50640ff020 2024-04-16T05:49:56.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.976+0000 7f5061600700 1 -- 172.21.15.27:0/3421747723 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+732 (secure 0 0 0) 0x7f505805aaa0 con 0x7f50640ff020 2024-04-16T05:49:56.974 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:49:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:49:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:49:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:49:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:49:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:49:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:49:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:49:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 5 2024-04-16T05:49:56.974 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:49:56.975 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:49:56.975 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4 2024-04-16T05:49:56.975 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:49:56.975 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:49:56.975 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 4, 2024-04-16T05:49:56.975 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 9 2024-04-16T05:49:56.975 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:49:56.975 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:49:56.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.979+0000 7f506c200700 1 -- 172.21.15.27:0/3421747723 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f504c06ec00 msgr2=0x7f504c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:56.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.979+0000 7f506c200700 1 --2- 172.21.15.27:0/3421747723 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f504c06ec00 0x7f504c0710b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5050000c00 tx=0x7f5050009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:56.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.979+0000 7f506c200700 1 -- 172.21.15.27:0/3421747723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50640ff020 msgr2=0x7f506410fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:56.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.979+0000 7f506c200700 1 --2- 172.21.15.27:0/3421747723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50640ff020 0x7f506410fb70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5058002a50 tx=0x7f50580046b0 comp rx=0 tx=0).stop 2024-04-16T05:49:56.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.979+0000 7f506c200700 1 -- 172.21.15.27:0/3421747723 shutdown_connections 2024-04-16T05:49:56.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.979+0000 7f506c200700 1 --2- 172.21.15.27:0/3421747723 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f504c06ec00 0x7f504c0710b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:56.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.979+0000 7f506c200700 1 --2- 172.21.15.27:0/3421747723 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50640ff020 0x7f506410fb70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:56.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.979+0000 7f506c200700 1 --2- 172.21.15.27:0/3421747723 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50641002d0 0x7f50641100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:56.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.979+0000 7f506c200700 1 -- 172.21.15.27:0/3421747723 >> 172.21.15.27:0/3421747723 conn(0x7f50640fa950 msgr2=0x7f50640fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:56.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.979+0000 7f506c200700 1 -- 172.21.15.27:0/3421747723 shutdown_connections 2024-04-16T05:49:56.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:56.979+0000 7f506c200700 1 -- 172.21.15.27:0/3421747723 wait complete. 2024-04-16T05:49:57.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.138+0000 7f1244000700 1 -- 172.21.15.27:0/168962936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f123c0fe0e0 msgr2=0x7f123c0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:57.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.138+0000 7f1244000700 1 --2- 172.21.15.27:0/168962936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f123c0fe0e0 0x7f123c0fe550 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1228004940 tx=0x7f1228009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:57.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.139+0000 7f1244000700 1 -- 172.21.15.27:0/168962936 shutdown_connections 2024-04-16T05:49:57.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.139+0000 7f1244000700 1 --2- 172.21.15.27:0/168962936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f123c0fe0e0 0x7f123c0fe550 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:57.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.139+0000 7f1244000700 1 --2- 172.21.15.27:0/168962936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f123c0fcef0 0x7f123c0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:57.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.139+0000 7f1244000700 1 -- 172.21.15.27:0/168962936 >> 172.21.15.27:0/168962936 conn(0x7f123c0f8530 msgr2=0x7f123c0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:57.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.139+0000 7f1244000700 1 -- 172.21.15.27:0/168962936 shutdown_connections 2024-04-16T05:49:57.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.139+0000 7f1244000700 1 -- 172.21.15.27:0/168962936 wait complete. 2024-04-16T05:49:57.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.139+0000 7f1244000700 1 Processor -- start 2024-04-16T05:49:57.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.140+0000 7f1244000700 1 -- start start 2024-04-16T05:49:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.140+0000 7f1244000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f123c0fcef0 0x7f123c10fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.140+0000 7f1244000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f123c0fe0e0 0x7f123c1101a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.140+0000 7f1244000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123c1107a0 con 0x7f123c0fe0e0 2024-04-16T05:49:57.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.140+0000 7f1244000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123c110ff0 con 0x7f123c0fcef0 2024-04-16T05:49:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.140+0000 7f1240c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f123c0fe0e0 0x7f123c1101a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.140+0000 7f1240c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f123c0fe0e0 0x7f123c1101a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42444/0 (socket says 172.21.15.27:42444) 2024-04-16T05:49:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.140+0000 7f1240c00700 1 -- 172.21.15.27:0/4069161283 learned_addr learned my addr 172.21.15.27:0/4069161283 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:49:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.141+0000 7f1241600700 1 --2- 172.21.15.27:0/4069161283 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f123c0fcef0 0x7f123c10fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.141+0000 7f1240c00700 1 -- 172.21.15.27:0/4069161283 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f123c0fcef0 msgr2=0x7f123c10fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.141+0000 7f1240c00700 1 --2- 172.21.15.27:0/4069161283 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f123c0fcef0 0x7f123c10fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.141+0000 7f1240c00700 1 -- 172.21.15.27:0/4069161283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1228005040 con 0x7f123c0fe0e0 2024-04-16T05:49:57.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.141+0000 7f1241600700 1 --2- 172.21.15.27:0/4069161283 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f123c0fcef0 0x7f123c10fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:49:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.141+0000 7f1240c00700 1 --2- 172.21.15.27:0/4069161283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f123c0fe0e0 0x7f123c1101a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1228009150 tx=0x7f1228009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.141+0000 7f1239600700 1 -- 172.21.15.27:0/4069161283 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f122800e040 con 0x7f123c0fe0e0 2024-04-16T05:49:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.141+0000 7f1239600700 1 -- 172.21.15.27:0/4069161283 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f122800bcc0 con 0x7f123c0fe0e0 2024-04-16T05:49:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.141+0000 7f1244000700 1 -- 172.21.15.27:0/4069161283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f123c111270 con 0x7f123c0fe0e0 2024-04-16T05:49:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.141+0000 7f1239600700 1 -- 172.21.15.27:0/4069161283 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f122801c630 con 0x7f123c0fe0e0 2024-04-16T05:49:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.142+0000 7f1244000700 1 -- 172.21.15.27:0/4069161283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f123c1117b0 con 0x7f123c0fe0e0 2024-04-16T05:49:57.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.143+0000 7f1244000700 1 -- 172.21.15.27:0/4069161283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f123c101a70 con 0x7f123c0fe0e0 2024-04-16T05:49:57.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.144+0000 7f1239600700 1 -- 172.21.15.27:0/4069161283 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1228013420 con 0x7f123c0fe0e0 2024-04-16T05:49:57.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.145+0000 7f1239600700 1 --2- 172.21.15.27:0/4069161283 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f122406ec00 0x7f12240710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:57.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.145+0000 7f1239600700 1 -- 172.21.15.27:0/4069161283 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f122808f6e0 con 0x7f123c0fe0e0 2024-04-16T05:49:57.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.145+0000 7f1241600700 1 --2- 172.21.15.27:0/4069161283 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f122406ec00 0x7f12240710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:57.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.146+0000 7f1241600700 1 --2- 172.21.15.27:0/4069161283 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f122406ec00 0x7f12240710b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f123c0fcc30 tx=0x7f1230009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:57.148 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.151+0000 7f1239600700 1 -- 172.21.15.27:0/4069161283 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f122805bed0 con 0x7f123c0fe0e0 2024-04-16T05:49:57.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.432+0000 7f1244000700 1 -- 172.21.15.27:0/4069161283 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f123c105af0 con 0x7f123c0fe0e0 2024-04-16T05:49:57.431 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.433+0000 7f1239600700 1 -- 172.21.15.27:0/4069161283 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1880 (secure 0 0 0) 0x7f122805ba60 con 0x7f123c0fe0e0 2024-04-16T05:49:57.432 INFO:teuthology.orchestra.run.smithi027.stdout:e16 2024-04-16T05:49:57.432 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:49:57.432 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:49:57.432 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:49:57.432 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:49:57.432 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:49:57.432 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:49:57.432 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 15 2024-04-16T05:49:57.432 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:49:57.433 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:49:57.433 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:40:32.701649+0000 2024-04-16T05:49:57.433 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:49:57.433 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:49:57.433 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:49:57.433 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:49:57.433 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:49:57.433 INFO:teuthology.orchestra.run.smithi027.stdout:max_xattr_size 65536 2024-04-16T05:49:57.433 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:49:57.433 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:49:57.433 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 46 2024-04-16T05:49:57.433 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:49:57.433 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:49:57.433 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:49:57.433 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=14462} 2024-04-16T05:49:57.434 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:49:57.434 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:49:57.434 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:49:57.434 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:49:57.434 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:49:57.434 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:49:57.434 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:49:57.434 INFO:teuthology.orchestra.run.smithi027.stdout:bal_rank_mask -1 2024-04-16T05:49:57.434 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:49:57.434 INFO:teuthology.orchestra.run.smithi027.stdout:qdb_cluster leader: 0 members: 2024-04-16T05:49:57.434 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:49:57.434 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:49:57.434 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:49:57.434 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:49:57.435 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:49:57.435 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{-1:14474} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2065690722,v1:172.21.15.27:6827/2065690722] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:49:57.435 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:24263} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:49:57.435 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:24273} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:49:57.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.436+0000 7f1244000700 1 -- 172.21.15.27:0/4069161283 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f122406ec00 msgr2=0x7f12240710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:57.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.436+0000 7f1244000700 1 --2- 172.21.15.27:0/4069161283 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f122406ec00 0x7f12240710b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f123c0fcc30 tx=0x7f1230009040 comp rx=0 tx=0).stop 2024-04-16T05:49:57.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.436+0000 7f1244000700 1 -- 172.21.15.27:0/4069161283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f123c0fe0e0 msgr2=0x7f123c1101a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:57.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.436+0000 7f1244000700 1 --2- 172.21.15.27:0/4069161283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f123c0fe0e0 0x7f123c1101a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1228009150 tx=0x7f1228009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:57.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.436+0000 7f1244000700 1 -- 172.21.15.27:0/4069161283 shutdown_connections 2024-04-16T05:49:57.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.436+0000 7f1244000700 1 --2- 172.21.15.27:0/4069161283 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f122406ec00 0x7f12240710b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:57.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.436+0000 7f1244000700 1 --2- 172.21.15.27:0/4069161283 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f123c0fcef0 0x7f123c10fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:57.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.436+0000 7f1244000700 1 --2- 172.21.15.27:0/4069161283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f123c0fe0e0 0x7f123c1101a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:57.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.436+0000 7f1244000700 1 -- 172.21.15.27:0/4069161283 >> 172.21.15.27:0/4069161283 conn(0x7f123c0f8530 msgr2=0x7f123c0f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:57.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.437+0000 7f1244000700 1 -- 172.21.15.27:0/4069161283 shutdown_connections 2024-04-16T05:49:57.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.437+0000 7f1244000700 1 -- 172.21.15.27:0/4069161283 wait complete. 2024-04-16T05:49:57.437 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2024-04-16T05:49:57.487 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:57 smithi122 podman[100112]: 2024-04-16 05:49:57.229358331 +0000 UTC m=+0.915126264 container remove 2123096d3d05b225280c8c7c5919d43b7244c00f7cb3d23b48255883e3b86eb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409) 2024-04-16T05:49:57.487 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:57 smithi122 systemd[1]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.5.service: Deactivated successfully. 2024-04-16T05:49:57.487 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:57 smithi122 systemd[1]: Stopped Ceph osd.5 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968. 2024-04-16T05:49:57.487 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:57 smithi122 systemd[1]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.5.service: Consumed 12.842s CPU time. 2024-04-16T05:49:57.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.590+0000 7f7a98e00700 1 -- 172.21.15.27:0/559796609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a940fff10 msgr2=0x7f7a94102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:57.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.590+0000 7f7a98e00700 1 --2- 172.21.15.27:0/559796609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a940fff10 0x7f7a94102330 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7a80004940 tx=0x7f7a80009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:57.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.590+0000 7f7a98e00700 1 -- 172.21.15.27:0/559796609 shutdown_connections 2024-04-16T05:49:57.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.590+0000 7f7a98e00700 1 --2- 172.21.15.27:0/559796609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a940fff10 0x7f7a94102330 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:57.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.590+0000 7f7a98e00700 1 --2- 172.21.15.27:0/559796609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a94066b80 0x7f7a940ff9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:57.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.590+0000 7f7a98e00700 1 -- 172.21.15.27:0/559796609 >> 172.21.15.27:0/559796609 conn(0x7f7a940f7440 msgr2=0x7f7a940f9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:57.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.590+0000 7f7a98e00700 1 -- 172.21.15.27:0/559796609 shutdown_connections 2024-04-16T05:49:57.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.590+0000 7f7a98e00700 1 -- 172.21.15.27:0/559796609 wait complete. 2024-04-16T05:49:57.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.591+0000 7f7a98e00700 1 Processor -- start 2024-04-16T05:49:57.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.591+0000 7f7a98e00700 1 -- start start 2024-04-16T05:49:57.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.591+0000 7f7a98e00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a94066b80 0x7f7a9410fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:57.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.592+0000 7f7a98e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a940fff10 0x7f7a941101a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:57.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.592+0000 7f7a98e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a941107a0 con 0x7f7a940fff10 2024-04-16T05:49:57.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.592+0000 7f7a98e00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a94110f80 con 0x7f7a94066b80 2024-04-16T05:49:57.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.592+0000 7f7a91400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a940fff10 0x7f7a941101a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:57.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.592+0000 7f7a91400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a940fff10 0x7f7a941101a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42458/0 (socket says 172.21.15.27:42458) 2024-04-16T05:49:57.590 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.592+0000 7f7a91400700 1 -- 172.21.15.27:0/2757933481 learned_addr learned my addr 172.21.15.27:0/2757933481 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:49:57.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.592+0000 7f7a91e00700 1 --2- 172.21.15.27:0/2757933481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a94066b80 0x7f7a9410fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:57.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.592+0000 7f7a91400700 1 -- 172.21.15.27:0/2757933481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a94066b80 msgr2=0x7f7a9410fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:57.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.592+0000 7f7a91400700 1 --2- 172.21.15.27:0/2757933481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a94066b80 0x7f7a9410fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:57.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.592+0000 7f7a91400700 1 -- 172.21.15.27:0/2757933481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a80005040 con 0x7f7a940fff10 2024-04-16T05:49:57.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.593+0000 7f7a91400700 1 --2- 172.21.15.27:0/2757933481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a940fff10 0x7f7a941101a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7a80009150 tx=0x7f7a80009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:57.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.593+0000 7f7a8a000700 1 -- 172.21.15.27:0/2757933481 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a8000e040 con 0x7f7a940fff10 2024-04-16T05:49:57.591 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.593+0000 7f7a8a000700 1 -- 172.21.15.27:0/2757933481 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a8000bbf0 con 0x7f7a940fff10 2024-04-16T05:49:57.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.593+0000 7f7a8a000700 1 -- 172.21.15.27:0/2757933481 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a8001c630 con 0x7f7a940fff10 2024-04-16T05:49:57.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.593+0000 7f7a98e00700 1 -- 172.21.15.27:0/2757933481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a94111200 con 0x7f7a940fff10 2024-04-16T05:49:57.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.593+0000 7f7a98e00700 1 -- 172.21.15.27:0/2757933481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a94111740 con 0x7f7a940fff10 2024-04-16T05:49:57.592 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.595+0000 7f7a98e00700 1 -- 172.21.15.27:0/2757933481 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a9405f5f0 con 0x7f7a940fff10 2024-04-16T05:49:57.593 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.596+0000 7f7a8a000700 1 -- 172.21.15.27:0/2757933481 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7a8000bd60 con 0x7f7a940fff10 2024-04-16T05:49:57.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.597+0000 7f7a8a000700 1 --2- 172.21.15.27:0/2757933481 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7a7806ec10 0x7f7a780710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:57.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.597+0000 7f7a8a000700 1 -- 172.21.15.27:0/2757933481 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f7a8008fc40 con 0x7f7a940fff10 2024-04-16T05:49:57.595 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.597+0000 7f7a91e00700 1 --2- 172.21.15.27:0/2757933481 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7a7806ec10 0x7f7a780710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:57.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.598+0000 7f7a91e00700 1 --2- 172.21.15.27:0/2757933481 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7a7806ec10 0x7f7a780710c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7a940668c0 tx=0x7f7a84009ee0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:57.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.602+0000 7f7a8a000700 1 -- 172.21.15.27:0/2757933481 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7a8005c570 con 0x7f7a940fff10 2024-04-16T05:49:57.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:57 smithi122 ceph-mon[80930]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:49:57.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:57 smithi122 ceph-mon[80930]: from='client.34374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:49:57.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:57 smithi122 ceph-mon[80930]: from='client.34378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:49:57.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:57 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/3421747723' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:49:57.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:57 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/4069161283' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:49:57.771 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:57 smithi122 systemd[1]: Starting Ceph osd.5 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T05:49:57.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:57 smithi027 ceph-mon[93635]: from='client.34370 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:49:57.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:57 smithi027 ceph-mon[93635]: from='client.34374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:49:57.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:57 smithi027 ceph-mon[93635]: from='client.34378 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:49:57.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:57 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/3421747723' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:49:57.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:57 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/4069161283' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:49:57.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.856+0000 7f7a98e00700 1 -- 172.21.15.27:0/2757933481 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a94111a70 con 0x7f7a7806ec10 2024-04-16T05:49:57.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.858+0000 7f7a8a000700 1 -- 172.21.15.27:0/2757933481 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+401 (secure 0 0 0) 0x7f7a94111a70 con 0x7f7a7806ec10 2024-04-16T05:49:57.856 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:49:57.856 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T05:49:57.856 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:49:57.856 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:49:57.856 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [ 2024-04-16T05:49:57.856 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr", 2024-04-16T05:49:57.856 INFO:teuthology.orchestra.run.smithi027.stdout: "mon", 2024-04-16T05:49:57.856 INFO:teuthology.orchestra.run.smithi027.stdout: "crash" 2024-04-16T05:49:57.856 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-16T05:49:57.856 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "11/21 daemons upgraded", 2024-04-16T05:49:57.857 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Currently upgrading osd daemons", 2024-04-16T05:49:57.857 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:49:57.857 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:49:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.860+0000 7f7a98e00700 1 -- 172.21.15.27:0/2757933481 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7a7806ec10 msgr2=0x7f7a780710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.860+0000 7f7a98e00700 1 --2- 172.21.15.27:0/2757933481 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7a7806ec10 0x7f7a780710c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7a940668c0 tx=0x7f7a84009ee0 comp rx=0 tx=0).stop 2024-04-16T05:49:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.861+0000 7f7a98e00700 1 -- 172.21.15.27:0/2757933481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a940fff10 msgr2=0x7f7a941101a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.861+0000 7f7a98e00700 1 --2- 172.21.15.27:0/2757933481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a940fff10 0x7f7a941101a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7a80009150 tx=0x7f7a80009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.861+0000 7f7a98e00700 1 -- 172.21.15.27:0/2757933481 shutdown_connections 2024-04-16T05:49:57.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.861+0000 7f7a98e00700 1 --2- 172.21.15.27:0/2757933481 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7a7806ec10 0x7f7a780710c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.861+0000 7f7a98e00700 1 --2- 172.21.15.27:0/2757933481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a94066b80 0x7f7a9410fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.861+0000 7f7a98e00700 1 --2- 172.21.15.27:0/2757933481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a940fff10 0x7f7a941101a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.861+0000 7f7a98e00700 1 -- 172.21.15.27:0/2757933481 >> 172.21.15.27:0/2757933481 conn(0x7f7a940f7440 msgr2=0x7f7a940fbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.861+0000 7f7a98e00700 1 -- 172.21.15.27:0/2757933481 shutdown_connections 2024-04-16T05:49:57.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:57.861+0000 7f7a98e00700 1 -- 172.21.15.27:0/2757933481 wait complete. 2024-04-16T05:49:58.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.014+0000 7f76e9c00700 1 -- 172.21.15.27:0/4276784639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76e40fee90 msgr2=0x7f76e40ff2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.014+0000 7f76e9c00700 1 --2- 172.21.15.27:0/4276784639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76e40fee90 0x7f76e40ff2a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f76d4004940 tx=0x7f76d4009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.014+0000 7f76e9c00700 1 -- 172.21.15.27:0/4276784639 shutdown_connections 2024-04-16T05:49:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.014+0000 7f76e9c00700 1 --2- 172.21.15.27:0/4276784639 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76e4100090 0x7f76e4100500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.014+0000 7f76e9c00700 1 --2- 172.21.15.27:0/4276784639 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76e40fee90 0x7f76e40ff2a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.015+0000 7f76e9c00700 1 -- 172.21.15.27:0/4276784639 >> 172.21.15.27:0/4276784639 conn(0x7f76e40fa790 msgr2=0x7f76e40fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.015+0000 7f76e9c00700 1 -- 172.21.15.27:0/4276784639 shutdown_connections 2024-04-16T05:49:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.015+0000 7f76e9c00700 1 -- 172.21.15.27:0/4276784639 wait complete. 2024-04-16T05:49:58.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.016+0000 7f76e9c00700 1 Processor -- start 2024-04-16T05:49:58.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.016+0000 7f76e9c00700 1 -- start start 2024-04-16T05:49:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.017+0000 7f76e9c00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76e40fee90 0x7f76e410fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.017+0000 7f76e9c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76e4100090 0x7f76e410ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.017+0000 7f76e9c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e41105a0 con 0x7f76e4100090 2024-04-16T05:49:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.017+0000 7f76e9c00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e4110df0 con 0x7f76e40fee90 2024-04-16T05:49:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.017+0000 7f76e3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76e4100090 0x7f76e410ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.017+0000 7f76e3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76e4100090 0x7f76e410ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42480/0 (socket says 172.21.15.27:42480) 2024-04-16T05:49:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.017+0000 7f76e3400700 1 -- 172.21.15.27:0/188855257 learned_addr learned my addr 172.21.15.27:0/188855257 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:49:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.017+0000 7f76e3e00700 1 --2- 172.21.15.27:0/188855257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76e40fee90 0x7f76e410fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.017+0000 7f76e3400700 1 -- 172.21.15.27:0/188855257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76e40fee90 msgr2=0x7f76e410fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.017+0000 7f76e3400700 1 --2- 172.21.15.27:0/188855257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76e40fee90 0x7f76e410fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.017+0000 7f76e3400700 1 -- 172.21.15.27:0/188855257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d4005040 con 0x7f76e4100090 2024-04-16T05:49:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.018+0000 7f76e3400700 1 --2- 172.21.15.27:0/188855257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76e4100090 0x7f76e410ffa0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f76d0002a50 tx=0x7f76d00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.018+0000 7f76dbe00700 1 -- 172.21.15.27:0/188855257 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76d0007cc0 con 0x7f76e4100090 2024-04-16T05:49:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.018+0000 7f76dbe00700 1 -- 172.21.15.27:0/188855257 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f76d000f040 con 0x7f76e4100090 2024-04-16T05:49:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.018+0000 7f76e9c00700 1 -- 172.21.15.27:0/188855257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e41110d0 con 0x7f76e4100090 2024-04-16T05:49:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.018+0000 7f76dbe00700 1 -- 172.21.15.27:0/188855257 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76d0009640 con 0x7f76e4100090 2024-04-16T05:49:58.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.018+0000 7f76e9c00700 1 -- 172.21.15.27:0/188855257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76e4111610 con 0x7f76e4100090 2024-04-16T05:49:58.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.019+0000 7f76e9c00700 1 -- 172.21.15.27:0/188855257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76e40fe290 con 0x7f76e4100090 2024-04-16T05:49:58.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.021+0000 7f76dbe00700 1 -- 172.21.15.27:0/188855257 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f76d00097a0 con 0x7f76e4100090 2024-04-16T05:49:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.022+0000 7f76dbe00700 1 --2- 172.21.15.27:0/188855257 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f76c806ec00 0x7f76c80710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:49:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.022+0000 7f76dbe00700 1 -- 172.21.15.27:0/188855257 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f76d008e7f0 con 0x7f76e4100090 2024-04-16T05:49:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.022+0000 7f76e3e00700 1 --2- 172.21.15.27:0/188855257 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f76c806ec00 0x7f76c80710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:49:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.023+0000 7f76e3e00700 1 --2- 172.21.15.27:0/188855257 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f76c806ec00 0x7f76c80710b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f76d400b3b0 tx=0x7f76d4009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:49:58.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.027+0000 7f76dbe00700 1 -- 172.21.15.27:0/188855257 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f76d005af60 con 0x7f76e4100090 2024-04-16T05:49:58.290 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:58 smithi122 podman[100213]: 2024-04-16 05:49:57.941201803 +0000 UTC m=+0.033012394 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T05:49:58.290 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:58 smithi122 podman[100213]: 2024-04-16 05:49:58.046874292 +0000 UTC m=+0.138684913 container create 818eee5f4c5e06760092d430d5e993f70bb4546d58b0bcd967ba4fda02935c93 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-16T05:49:58.290 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:58 smithi122 podman[100213]: 2024-04-16 05:49:58.292494949 +0000 UTC m=+0.384305610 container init 818eee5f4c5e06760092d430d5e993f70bb4546d58b0bcd967ba4fda02935c93 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2) 2024-04-16T05:49:58.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.348+0000 7f76e9c00700 1 -- 172.21.15.27:0/188855257 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f76e4111990 con 0x7f76e4100090 2024-04-16T05:49:58.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.348+0000 7f76dbe00700 1 -- 172.21.15.27:0/188855257 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+883 (secure 0 0 0) 0x7f76d005aaf0 con 0x7f76e4100090 2024-04-16T05:49:58.346 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Reduced data availability: 1 pg inactive, 1 pg peering; Degraded data redundancy: 6/72 objects degraded (8.333%), 3 pgs degraded 2024-04-16T05:49:58.346 INFO:teuthology.orchestra.run.smithi027.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-16T05:49:58.346 INFO:teuthology.orchestra.run.smithi027.stdout: osd.5 (root=default,host=smithi122) is down 2024-04-16T05:49:58.347 INFO:teuthology.orchestra.run.smithi027.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-04-16T05:49:58.347 INFO:teuthology.orchestra.run.smithi027.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-04-16T05:49:58.347 INFO:teuthology.orchestra.run.smithi027.stdout:[WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg peering 2024-04-16T05:49:58.347 INFO:teuthology.orchestra.run.smithi027.stdout: pg 2.2 is stuck peering for 119s, current state peering, last acting [1,0] 2024-04-16T05:49:58.347 INFO:teuthology.orchestra.run.smithi027.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 6/72 objects degraded (8.333%), 3 pgs degraded 2024-04-16T05:49:58.347 INFO:teuthology.orchestra.run.smithi027.stdout: pg 2.b is active+undersized+degraded, acting [3,4] 2024-04-16T05:49:58.347 INFO:teuthology.orchestra.run.smithi027.stdout: pg 2.14 is active+undersized+degraded, acting [3,4] 2024-04-16T05:49:58.347 INFO:teuthology.orchestra.run.smithi027.stdout: pg 2.1d is active+undersized+degraded, acting [3,0] 2024-04-16T05:49:58.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.351+0000 7f76e9c00700 1 -- 172.21.15.27:0/188855257 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f76c806ec00 msgr2=0x7f76c80710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:58.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.351+0000 7f76e9c00700 1 --2- 172.21.15.27:0/188855257 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f76c806ec00 0x7f76c80710b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f76d400b3b0 tx=0x7f76d4009e30 comp rx=0 tx=0).stop 2024-04-16T05:49:58.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.351+0000 7f76e9c00700 1 -- 172.21.15.27:0/188855257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76e4100090 msgr2=0x7f76e410ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:49:58.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.351+0000 7f76e9c00700 1 --2- 172.21.15.27:0/188855257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76e4100090 0x7f76e410ffa0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f76d0002a50 tx=0x7f76d00046b0 comp rx=0 tx=0).stop 2024-04-16T05:49:58.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.351+0000 7f76e9c00700 1 -- 172.21.15.27:0/188855257 shutdown_connections 2024-04-16T05:49:58.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.351+0000 7f76e9c00700 1 --2- 172.21.15.27:0/188855257 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f76c806ec00 0x7f76c80710b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:58.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.351+0000 7f76e9c00700 1 --2- 172.21.15.27:0/188855257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76e40fee90 0x7f76e410fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:58.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.351+0000 7f76e9c00700 1 --2- 172.21.15.27:0/188855257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76e4100090 0x7f76e410ffa0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:49:58.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.351+0000 7f76e9c00700 1 -- 172.21.15.27:0/188855257 >> 172.21.15.27:0/188855257 conn(0x7f76e40fa790 msgr2=0x7f76e40fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:49:58.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.351+0000 7f76e9c00700 1 -- 172.21.15.27:0/188855257 shutdown_connections 2024-04-16T05:49:58.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:49:58.351+0000 7f76e9c00700 1 -- 172.21.15.27:0/188855257 wait complete. 2024-04-16T05:49:58.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:58 smithi122 ceph-mon[80930]: pgmap v168: 65 pgs: 3 active+undersized+degraded, 9 peering, 18 active+undersized, 35 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail; 6/72 objects degraded (8.333%) 2024-04-16T05:49:58.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:58 smithi122 ceph-mon[80930]: Health check failed: Degraded data redundancy: 6/72 objects degraded (8.333%), 3 pgs degraded (PG_DEGRADED) 2024-04-16T05:49:58.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:58 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/188855257' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:49:58.772 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:58 smithi122 podman[100213]: 2024-04-16 05:49:58.303387465 +0000 UTC m=+0.395198096 container start 818eee5f4c5e06760092d430d5e993f70bb4546d58b0bcd967ba4fda02935c93 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-16T05:49:58.772 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:58 smithi122 podman[100213]: 2024-04-16 05:49:58.391456036 +0000 UTC m=+0.483266691 container attach 818eee5f4c5e06760092d430d5e993f70bb4546d58b0bcd967ba4fda02935c93 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-activate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-16T05:49:58.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:58 smithi027 ceph-mon[93635]: pgmap v168: 65 pgs: 3 active+undersized+degraded, 9 peering, 18 active+undersized, 35 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail; 6/72 objects degraded (8.333%) 2024-04-16T05:49:58.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:58 smithi027 ceph-mon[93635]: Health check failed: Degraded data redundancy: 6/72 objects degraded (8.333%), 3 pgs degraded (PG_DEGRADED) 2024-04-16T05:49:58.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:58 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/188855257' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:49:59.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 464, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 100302 2024-04-16T05:49:59.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 464, success, outsize: 120 2024-04-16T05:49:59.494 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:59 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-activate[100227]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-16T05:49:59.494 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:59 smithi122 bash[100213]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-16T05:49:59.494 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:59 smithi122 bash[100213]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-16T05:49:59.494 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:59 smithi122 bash[100213]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-16T05:49:59.494 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:59 smithi122 bash[100213]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-16T05:49:59.494 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:59 smithi122 bash[100213]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-16T05:49:59.494 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:59 smithi122 bash[100213]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-16T05:49:59.494 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:59 smithi122 bash[100213]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-16T05:49:59.495 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:59 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-activate[100227]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-16T05:49:59.495 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:59 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-activate[100227]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-16T05:49:59.495 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:59 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-activate[100227]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-16T05:49:59.495 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:59 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-activate[100227]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5/block 2024-04-16T05:49:59.495 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:59 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-activate[100227]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-16T05:49:59.495 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:59 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-activate[100227]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-16T05:49:59.495 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:59 smithi122 podman[100213]: 2024-04-16 05:49:59.255235782 +0000 UTC m=+1.347046400 container died 818eee5f4c5e06760092d430d5e993f70bb4546d58b0bcd967ba4fda02935c93 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-activate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , ceph=True) 2024-04-16T05:49:59.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:59 smithi122 ceph-mon[80930]: from='client.34390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:49:59.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:59 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:59.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:49:59 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:49:59.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:59 smithi027 ceph-mon[93635]: from='client.34390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:49:59.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:59 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:49:59.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:49:59 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:50:00.271 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:49:59 smithi122 podman[100346]: 2024-04-16 05:49:59.944554892 +0000 UTC m=+0.667660348 container remove 818eee5f4c5e06760092d430d5e993f70bb4546d58b0bcd967ba4fda02935c93 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-activate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True) 2024-04-16T05:50:00.547 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:50:00 smithi122 podman[100406]: 2024-04-16 05:50:00.201730901 +0000 UTC m=+0.033268321 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T05:50:00.547 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:50:00 smithi122 podman[100406]: 2024-04-16 05:50:00.312157008 +0000 UTC m=+0.143694508 container create fd12260a820d761235c1d30eb3f0b5e36e57d14af4179de7dc0b0ad795477229 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-16T05:50:00.547 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:50:00 smithi122 podman[100406]: 2024-04-16 05:50:00.548929733 +0000 UTC m=+0.380467196 container init fd12260a820d761235c1d30eb3f0b5e36e57d14af4179de7dc0b0ad795477229 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5, ceph=True, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T05:50:00.547 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:00 smithi122 ceph-mon[80930]: pgmap v169: 65 pgs: 5 active+undersized+degraded, 25 active+undersized, 35 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T05:50:00.547 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:00 smithi122 ceph-mon[80930]: Health detail: HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Reduced data availability: 1 pg inactive, 1 pg peering; Degraded data redundancy: 6/72 objects degraded (8.333%), 3 pgs degraded 2024-04-16T05:50:00.547 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:00 smithi122 ceph-mon[80930]: [WRN] OSD_DOWN: 1 osds down 2024-04-16T05:50:00.547 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:00 smithi122 ceph-mon[80930]: osd.5 (root=default,host=smithi122) is down 2024-04-16T05:50:00.548 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:00 smithi122 ceph-mon[80930]: [WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-04-16T05:50:00.548 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:00 smithi122 ceph-mon[80930]: all OSDs are running squid or later but require_osd_release < squid 2024-04-16T05:50:00.548 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:00 smithi122 ceph-mon[80930]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg peering 2024-04-16T05:50:00.548 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:00 smithi122 ceph-mon[80930]: pg 2.2 is stuck peering for 119s, current state peering, last acting [1,0] 2024-04-16T05:50:00.548 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:00 smithi122 ceph-mon[80930]: [WRN] PG_DEGRADED: Degraded data redundancy: 6/72 objects degraded (8.333%), 3 pgs degraded 2024-04-16T05:50:00.548 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:00 smithi122 ceph-mon[80930]: pg 2.b is active+undersized+degraded, acting [3,4] 2024-04-16T05:50:00.548 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:00 smithi122 ceph-mon[80930]: pg 2.14 is active+undersized+degraded, acting [3,4] 2024-04-16T05:50:00.548 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:00 smithi122 ceph-mon[80930]: pg 2.1d is active+undersized+degraded, acting [3,0] 2024-04-16T05:50:00.548 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:00 smithi122 ceph-mon[80930]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-16T05:50:00.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:00 smithi027 ceph-mon[93635]: pgmap v169: 65 pgs: 5 active+undersized+degraded, 25 active+undersized, 35 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T05:50:00.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:00 smithi027 ceph-mon[93635]: Health detail: HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Reduced data availability: 1 pg inactive, 1 pg peering; Degraded data redundancy: 6/72 objects degraded (8.333%), 3 pgs degraded 2024-04-16T05:50:00.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:00 smithi027 ceph-mon[93635]: [WRN] OSD_DOWN: 1 osds down 2024-04-16T05:50:00.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:00 smithi027 ceph-mon[93635]: osd.5 (root=default,host=smithi122) is down 2024-04-16T05:50:00.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:00 smithi027 ceph-mon[93635]: [WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-04-16T05:50:00.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:00 smithi027 ceph-mon[93635]: all OSDs are running squid or later but require_osd_release < squid 2024-04-16T05:50:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:00 smithi027 ceph-mon[93635]: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg peering 2024-04-16T05:50:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:00 smithi027 ceph-mon[93635]: pg 2.2 is stuck peering for 119s, current state peering, last acting [1,0] 2024-04-16T05:50:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:00 smithi027 ceph-mon[93635]: [WRN] PG_DEGRADED: Degraded data redundancy: 6/72 objects degraded (8.333%), 3 pgs degraded 2024-04-16T05:50:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:00 smithi027 ceph-mon[93635]: pg 2.b is active+undersized+degraded, acting [3,4] 2024-04-16T05:50:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:00 smithi027 ceph-mon[93635]: pg 2.14 is active+undersized+degraded, acting [3,4] 2024-04-16T05:50:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:00 smithi027 ceph-mon[93635]: pg 2.1d is active+undersized+degraded, acting [3,0] 2024-04-16T05:50:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:00 smithi027 ceph-mon[93635]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-16T05:50:01.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:50:00 smithi122 podman[100406]: 2024-04-16 05:50:00.561720089 +0000 UTC m=+0.393257519 container start fd12260a820d761235c1d30eb3f0b5e36e57d14af4179de7dc0b0ad795477229 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-16T05:50:01.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:50:00 smithi122 ceph-osd[100425]: -- 172.21.15.122:0/1897117433 <== mon.1 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557e03385680 con 0x557e0334ec00 2024-04-16T05:50:01.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:50:00 smithi122 bash[100406]: fd12260a820d761235c1d30eb3f0b5e36e57d14af4179de7dc0b0ad795477229 2024-04-16T05:50:01.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:50:00 smithi122 systemd[1]: Started Ceph osd.5 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968. 2024-04-16T05:50:01.521 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:50:01 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T05:50:01.137+0000 7f89a59e57c0 -1 Falling back to public interface 2024-04-16T05:50:02.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:01 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:02.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:01 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:02.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:01 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:50:02.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:01 smithi122 ceph-mon[80930]: pgmap v170: 65 pgs: 5 active+undersized+degraded, 25 active+undersized, 35 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T05:50:02.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:01 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:02.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:01 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:02.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:01 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:50:02.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:01 smithi027 ceph-mon[93635]: pgmap v170: 65 pgs: 5 active+undersized+degraded, 25 active+undersized, 35 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T05:50:04.442 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:04 smithi122 ceph-mon[80930]: pgmap v171: 65 pgs: 5 active+undersized+degraded, 25 active+undersized, 35 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T05:50:04.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:04 smithi027 ceph-mon[93635]: pgmap v171: 65 pgs: 5 active+undersized+degraded, 25 active+undersized, 35 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T05:50:05.069 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:50:04 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T05:50:04.795+0000 7f89a59e57c0 -1 osd.5 78 log_to_monitors true 2024-04-16T05:50:05.336 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:05 smithi122 ceph-mon[80930]: Health check update: Degraded data redundancy: 8/72 objects degraded (11.111%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T05:50:05.336 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:05 smithi122 ceph-mon[80930]: from='osd.5 [v2:172.21.15.122:6816/3634763653,v1:172.21.15.122:6817/3634763653]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T05:50:05.336 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:05 smithi122 ceph-mon[80930]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T05:50:05.336 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:05 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:05.336 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:05 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:05.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:05 smithi027 ceph-mon[93635]: Health check update: Degraded data redundancy: 8/72 objects degraded (11.111%), 5 pgs degraded (PG_DEGRADED) 2024-04-16T05:50:05.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:05 smithi027 ceph-mon[93635]: from='osd.5 [v2:172.21.15.122:6816/3634763653,v1:172.21.15.122:6817/3634763653]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T05:50:05.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:05 smithi027 ceph-mon[93635]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-16T05:50:05.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:05 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:05.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:05 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:06.324 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:06 smithi122 ceph-mon[80930]: pgmap v172: 65 pgs: 5 active+undersized+degraded, 25 active+undersized, 35 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T05:50:06.324 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:06 smithi122 ceph-mon[80930]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-16T05:50:06.324 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:06 smithi122 ceph-mon[80930]: osdmap e81: 6 total, 5 up, 6 in 2024-04-16T05:50:06.324 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:06 smithi122 ceph-mon[80930]: from='osd.5 [v2:172.21.15.122:6816/3634763653,v1:172.21.15.122:6817/3634763653]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:50:06.324 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:06 smithi122 ceph-mon[80930]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:50:06.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:06 smithi027 ceph-mon[93635]: pgmap v172: 65 pgs: 5 active+undersized+degraded, 25 active+undersized, 35 active+clean; 584 KiB data, 431 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T05:50:06.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:06 smithi027 ceph-mon[93635]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-16T05:50:06.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:06 smithi027 ceph-mon[93635]: osdmap e81: 6 total, 5 up, 6 in 2024-04-16T05:50:06.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:06 smithi027 ceph-mon[93635]: from='osd.5 [v2:172.21.15.122:6816/3634763653,v1:172.21.15.122:6817/3634763653]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:50:06.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:06 smithi027 ceph-mon[93635]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2024-04-16T05:50:06.771 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 05:50:06 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T05:50:06.325+0000 7f898fe00700 -1 osd.5 78 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-16T05:50:07.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:07 smithi122 ceph-mon[80930]: from='osd.5 ' entity='osd.5' 2024-04-16T05:50:07.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:07 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:07.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:07 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:07.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:07 smithi027 ceph-mon[93635]: from='osd.5 ' entity='osd.5' 2024-04-16T05:50:07.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:07 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:07.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:07 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:08.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:08 smithi122 ceph-mon[80930]: pgmap v174: 65 pgs: 5 active+undersized+degraded, 25 active+undersized, 35 active+clean; 584 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T05:50:08.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:08 smithi122 ceph-mon[80930]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T05:50:08.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:08 smithi122 ceph-mon[80930]: osd.5 [v2:172.21.15.122:6816/3634763653,v1:172.21.15.122:6817/3634763653] boot 2024-04-16T05:50:08.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:08 smithi122 ceph-mon[80930]: osdmap e82: 6 total, 6 up, 6 in 2024-04-16T05:50:08.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:08 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T05:50:08.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:08 smithi027 ceph-mon[93635]: pgmap v174: 65 pgs: 5 active+undersized+degraded, 25 active+undersized, 35 active+clean; 584 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 8/72 objects degraded (11.111%) 2024-04-16T05:50:08.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:08 smithi027 ceph-mon[93635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-16T05:50:08.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:08 smithi027 ceph-mon[93635]: osd.5 [v2:172.21.15.122:6816/3634763653,v1:172.21.15.122:6817/3634763653] boot 2024-04-16T05:50:08.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:08 smithi027 ceph-mon[93635]: osdmap e82: 6 total, 6 up, 6 in 2024-04-16T05:50:08.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:08 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-16T05:50:09.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:09 smithi122 ceph-mon[80930]: osdmap e83: 6 total, 6 up, 6 in 2024-04-16T05:50:09.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:09 smithi027 ceph-mon[93635]: osdmap e83: 6 total, 6 up, 6 in 2024-04-16T05:50:10.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:10 smithi122 ceph-mon[80930]: pgmap v177: 65 pgs: 3 active+undersized+degraded, 22 active+undersized, 40 active+clean; 584 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 6/72 objects degraded (8.333%) 2024-04-16T05:50:10.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:10 smithi122 ceph-mon[80930]: Health check update: Degraded data redundancy: 6/72 objects degraded (8.333%), 3 pgs degraded (PG_DEGRADED) 2024-04-16T05:50:10.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:10 smithi027 ceph-mon[93635]: pgmap v177: 65 pgs: 3 active+undersized+degraded, 22 active+undersized, 40 active+clean; 584 KiB data, 168 MiB used, 536 GiB / 536 GiB avail; 6/72 objects degraded (8.333%) 2024-04-16T05:50:10.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:10 smithi027 ceph-mon[93635]: Health check update: Degraded data redundancy: 6/72 objects degraded (8.333%), 3 pgs degraded (PG_DEGRADED) 2024-04-16T05:50:10.955 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 466, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102132 2024-04-16T05:50:10.956 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 466, success, outsize: 120 2024-04-16T05:50:10.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 468, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102210 2024-04-16T05:50:10.981 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 468, success, outsize: 120 2024-04-16T05:50:11.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 470, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102291 2024-04-16T05:50:11.006 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 470, success, outsize: 120 2024-04-16T05:50:11.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 472, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102378 2024-04-16T05:50:11.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 472, success, outsize: 120 2024-04-16T05:50:11.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 474, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102462 2024-04-16T05:50:11.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 474, success, outsize: 120 2024-04-16T05:50:11.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 476, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102540 2024-04-16T05:50:11.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 476, success, outsize: 120 2024-04-16T05:50:11.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 478, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102621 2024-04-16T05:50:11.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 478, success, outsize: 120 2024-04-16T05:50:11.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 480, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102708 2024-04-16T05:50:11.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 480, success, outsize: 120 2024-04-16T05:50:11.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 482, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102792 2024-04-16T05:50:11.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 482, success, outsize: 120 2024-04-16T05:50:11.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 484, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102870 2024-04-16T05:50:11.151 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 484, success, outsize: 120 2024-04-16T05:50:11.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 486, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 102951 2024-04-16T05:50:11.167 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 486, success, outsize: 120 2024-04-16T05:50:11.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 488, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103038 2024-04-16T05:50:11.184 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 488, success, outsize: 120 2024-04-16T05:50:11.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 490, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103122 2024-04-16T05:50:11.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 490, success, outsize: 120 2024-04-16T05:50:11.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 492, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103200 2024-04-16T05:50:11.219 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 492, success, outsize: 120 2024-04-16T05:50:11.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 494, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103281 2024-04-16T05:50:11.237 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 494, success, outsize: 120 2024-04-16T05:50:11.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 496, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103368 2024-04-16T05:50:11.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 496, success, outsize: 120 2024-04-16T05:50:11.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 498, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103452 2024-04-16T05:50:11.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 498, success, outsize: 120 2024-04-16T05:50:11.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 500, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103530 2024-04-16T05:50:11.293 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 500, success, outsize: 120 2024-04-16T05:50:11.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 502, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103611 2024-04-16T05:50:11.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 502, success, outsize: 120 2024-04-16T05:50:11.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 504, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103698 2024-04-16T05:50:11.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 504, success, outsize: 120 2024-04-16T05:50:11.353 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 506, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103782 2024-04-16T05:50:11.354 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 506, success, outsize: 120 2024-04-16T05:50:11.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 508, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103860 2024-04-16T05:50:11.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 508, success, outsize: 120 2024-04-16T05:50:11.393 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 510, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 103941 2024-04-16T05:50:11.394 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 510, success, outsize: 120 2024-04-16T05:50:11.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 512, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104028 2024-04-16T05:50:11.416 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 512, success, outsize: 120 2024-04-16T05:50:11.437 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 514, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104112 2024-04-16T05:50:11.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 514, success, outsize: 120 2024-04-16T05:50:11.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 516, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104190 2024-04-16T05:50:11.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 516, success, outsize: 120 2024-04-16T05:50:11.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 518, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104271 2024-04-16T05:50:11.472 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 518, success, outsize: 120 2024-04-16T05:50:11.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 520, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104358 2024-04-16T05:50:11.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 520, success, outsize: 120 2024-04-16T05:50:12.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:12 smithi122 ceph-mon[80930]: pgmap v178: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 6/72 objects degraded (8.333%) 2024-04-16T05:50:12.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:12 smithi027 ceph-mon[93635]: pgmap v178: 65 pgs: 3 active+undersized+degraded, 6 active+undersized, 56 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 6/72 objects degraded (8.333%) 2024-04-16T05:50:13.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:13.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:13.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:50:13.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:50:13.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:13.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:50:13.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:50:13.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:50:13.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:50:13.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:50:13.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:13.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-16T05:50:13.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-16T05:50:13.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-16T05:50:13.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-16T05:50:13.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-16T05:50:13.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-16T05:50:13.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-16T05:50:13.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-16T05:50:13.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-16T05:50:13.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-16T05:50:13.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-16T05:50:13.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-16T05:50:13.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-16T05:50:13.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:13.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:13.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:50:13.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:50:13.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:13.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:50:13.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:50:13.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:50:13.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:50:13.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:50:13.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:13.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-16T05:50:13.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-16T05:50:13.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-16T05:50:13.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-16T05:50:13.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-16T05:50:13.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-16T05:50:13.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-16T05:50:13.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-16T05:50:13.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-16T05:50:13.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-16T05:50:13.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-16T05:50:13.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-16T05:50:13.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-16T05:50:14.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:14 smithi122 ceph-mon[80930]: Upgrade: Setting container_image for all osd 2024-04-16T05:50:14.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:14 smithi122 ceph-mon[80930]: Upgrade: Setting require_osd_release to 19 squid 2024-04-16T05:50:14.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:14 smithi122 ceph-mon[80930]: pgmap v179: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:14.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:14 smithi122 ceph-mon[80930]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/72 objects degraded (8.333%), 3 pgs degraded) 2024-04-16T05:50:14.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:14 smithi122 ceph-mon[80930]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-16T05:50:14.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-16T05:50:14.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:14 smithi122 ceph-mon[80930]: osdmap e84: 6 total, 6 up, 6 in 2024-04-16T05:50:14.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi027.hkhpuq"]}]: dispatch 2024-04-16T05:50:14.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:14.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:50:14.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:14 smithi027 ceph-mon[93635]: Upgrade: Setting container_image for all osd 2024-04-16T05:50:14.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:14 smithi027 ceph-mon[93635]: Upgrade: Setting require_osd_release to 19 squid 2024-04-16T05:50:14.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:14 smithi027 ceph-mon[93635]: pgmap v179: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:14.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:14 smithi027 ceph-mon[93635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/72 objects degraded (8.333%), 3 pgs degraded) 2024-04-16T05:50:14.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:14 smithi027 ceph-mon[93635]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-16T05:50:14.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-16T05:50:14.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:14 smithi027 ceph-mon[93635]: osdmap e84: 6 total, 6 up, 6 in 2024-04-16T05:50:14.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi027.hkhpuq"]}]: dispatch 2024-04-16T05:50:14.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:14.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:50:15.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:15 smithi027 ceph-mon[93635]: Upgrade: It appears safe to stop mds.cephfs.smithi027.hkhpuq 2024-04-16T05:50:15.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:15 smithi122 ceph-mon[80930]: Upgrade: It appears safe to stop mds.cephfs.smithi027.hkhpuq 2024-04-16T05:50:16.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:16 smithi122 ceph-mon[80930]: pgmap v181: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:16.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:16 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:16.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:16 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.hkhpuq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:50:16.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:16 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:50:16.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:16 smithi027 ceph-mon[93635]: pgmap v181: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:16.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:16 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:16.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:16 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.hkhpuq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:50:16.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:16 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:50:17.524 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:17 smithi027 ceph-mon[93635]: Upgrade: Updating mds.cephfs.smithi027.hkhpuq 2024-04-16T05:50:17.524 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:17 smithi027 ceph-mon[93635]: Deploying daemon mds.cephfs.smithi027.hkhpuq on smithi027 2024-04-16T05:50:17.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:17 smithi122 ceph-mon[80930]: Upgrade: Updating mds.cephfs.smithi027.hkhpuq 2024-04-16T05:50:17.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:17 smithi122 ceph-mon[80930]: Deploying daemon mds.cephfs.smithi027.hkhpuq on smithi027 2024-04-16T05:50:18.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:18 smithi122 ceph-mon[80930]: pgmap v182: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:18.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:18 smithi027 ceph-mon[93635]: pgmap v182: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:20.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:20 smithi122 ceph-mon[80930]: pgmap v183: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:20.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:20 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:20.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:20 smithi122 ceph-mon[80930]: osdmap e85: 6 total, 6 up, 6 in 2024-04-16T05:50:20.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:20 smithi027 ceph-mon[93635]: pgmap v183: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:20.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:20 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:20.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:20 smithi027 ceph-mon[93635]: osdmap e85: 6 total, 6 up, 6 in 2024-04-16T05:50:21.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:21 smithi122 ceph-mon[80930]: fsmap cephfs:1 {0=cephfs.smithi027.ejoytk=up:active} 2 up:standby 2024-04-16T05:50:21.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:21 smithi027 ceph-mon[93635]: fsmap cephfs:1 {0=cephfs.smithi027.ejoytk=up:active} 2 up:standby 2024-04-16T05:50:22.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:22 smithi122 ceph-mon[80930]: pgmap v185: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:22.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:22 smithi027 ceph-mon[93635]: pgmap v185: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:24.270 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:24 smithi027 ceph-mon[93635]: pgmap v186: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:24.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:24 smithi122 ceph-mon[80930]: pgmap v186: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:25.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:25 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:25.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:25 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:25.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:25 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:50:25.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:25 smithi122 ceph-mon[80930]: mds.? [v2:172.21.15.27:6826/2831468256,v1:172.21.15.27:6827/2831468256] up:boot 2024-04-16T05:50:25.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:25 smithi122 ceph-mon[80930]: fsmap cephfs:1 {0=cephfs.smithi027.ejoytk=up:active} 3 up:standby 2024-04-16T05:50:25.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:25 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.hkhpuq"}]: dispatch 2024-04-16T05:50:25.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:25 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:25.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:25 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:25.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:25 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:50:25.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:25 smithi027 ceph-mon[93635]: mds.? [v2:172.21.15.27:6826/2831468256,v1:172.21.15.27:6827/2831468256] up:boot 2024-04-16T05:50:25.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:25 smithi027 ceph-mon[93635]: fsmap cephfs:1 {0=cephfs.smithi027.ejoytk=up:active} 3 up:standby 2024-04-16T05:50:25.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:25 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.hkhpuq"}]: dispatch 2024-04-16T05:50:26.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:26 smithi122 ceph-mon[80930]: pgmap v187: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:26.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:26 smithi027 ceph-mon[93635]: pgmap v187: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:28.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.533+0000 7f498e200700 1 -- 172.21.15.27:0/945409030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49880fcef0 msgr2=0x7f49880fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:28.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.533+0000 7f498e200700 1 --2- 172.21.15.27:0/945409030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49880fcef0 0x7f49880fd320 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4978004940 tx=0x7f4978009e30 comp rx=0 tx=0).stop 2024-04-16T05:50:28.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.533+0000 7f498e200700 1 -- 172.21.15.27:0/945409030 shutdown_connections 2024-04-16T05:50:28.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.533+0000 7f498e200700 1 --2- 172.21.15.27:0/945409030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49880fe0e0 0x7f49880fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:28.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.533+0000 7f498e200700 1 --2- 172.21.15.27:0/945409030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49880fcef0 0x7f49880fd320 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:28.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.533+0000 7f498e200700 1 -- 172.21.15.27:0/945409030 >> 172.21.15.27:0/945409030 conn(0x7f49880f8530 msgr2=0x7f49880fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:50:28.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.533+0000 7f498e200700 1 -- 172.21.15.27:0/945409030 shutdown_connections 2024-04-16T05:50:28.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.533+0000 7f498e200700 1 -- 172.21.15.27:0/945409030 wait complete. 2024-04-16T05:50:28.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.534+0000 7f498e200700 1 Processor -- start 2024-04-16T05:50:28.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.534+0000 7f498e200700 1 -- start start 2024-04-16T05:50:28.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.535+0000 7f498e200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49880fcef0 0x7f498810fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:50:28.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.535+0000 7f498e200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49880fe0e0 0x7f4988110290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:50:28.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.535+0000 7f498e200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4988110890 con 0x7f49880fe0e0 2024-04-16T05:50:28.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.535+0000 7f498e200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4988110fd0 con 0x7f49880fcef0 2024-04-16T05:50:28.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.535+0000 7f4986a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49880fe0e0 0x7f4988110290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:50:28.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.535+0000 7f4986a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49880fe0e0 0x7f4988110290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37300/0 (socket says 172.21.15.27:37300) 2024-04-16T05:50:28.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.535+0000 7f4986a00700 1 -- 172.21.15.27:0/875282751 learned_addr learned my addr 172.21.15.27:0/875282751 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:50:28.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.535+0000 7f4987400700 1 --2- 172.21.15.27:0/875282751 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49880fcef0 0x7f498810fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:50:28.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.536+0000 7f4986a00700 1 -- 172.21.15.27:0/875282751 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49880fcef0 msgr2=0x7f498810fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:28.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.536+0000 7f4986a00700 1 --2- 172.21.15.27:0/875282751 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49880fcef0 0x7f498810fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:28.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.536+0000 7f4986a00700 1 -- 172.21.15.27:0/875282751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4978005040 con 0x7f49880fe0e0 2024-04-16T05:50:28.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.536+0000 7f4986a00700 1 --2- 172.21.15.27:0/875282751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49880fe0e0 0x7f4988110290 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f497400c810 tx=0x7f4974004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:50:28.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.537+0000 7f497f400700 1 -- 172.21.15.27:0/875282751 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4974012810 con 0x7f49880fe0e0 2024-04-16T05:50:28.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.537+0000 7f497f400700 1 -- 172.21.15.27:0/875282751 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4974012e40 con 0x7f49880fe0e0 2024-04-16T05:50:28.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.537+0000 7f497f400700 1 -- 172.21.15.27:0/875282751 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49740102e0 con 0x7f49880fe0e0 2024-04-16T05:50:28.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.537+0000 7f498e200700 1 -- 172.21.15.27:0/875282751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49881112b0 con 0x7f49880fe0e0 2024-04-16T05:50:28.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.537+0000 7f498e200700 1 -- 172.21.15.27:0/875282751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4988111870 con 0x7f49880fe0e0 2024-04-16T05:50:28.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.538+0000 7f498e200700 1 -- 172.21.15.27:0/875282751 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4988101a70 con 0x7f49880fe0e0 2024-04-16T05:50:28.537 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.540+0000 7f497f400700 1 -- 172.21.15.27:0/875282751 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4974012970 con 0x7f49880fe0e0 2024-04-16T05:50:28.538 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.541+0000 7f497f400700 1 --2- 172.21.15.27:0/875282751 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f496c06ec00 0x7f496c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:50:28.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.541+0000 7f497f400700 1 -- 172.21.15.27:0/875282751 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f497408ec60 con 0x7f49880fe0e0 2024-04-16T05:50:28.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.541+0000 7f4987400700 1 --2- 172.21.15.27:0/875282751 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f496c06ec00 0x7f496c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:50:28.539 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.542+0000 7f4987400700 1 --2- 172.21.15.27:0/875282751 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f496c06ec00 0x7f496c0710b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f497800b970 tx=0x7f4978009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:50:28.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.551+0000 7f497f400700 1 -- 172.21.15.27:0/875282751 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f497405b4d0 con 0x7f49880fe0e0 2024-04-16T05:50:28.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:28 smithi122 ceph-mon[80930]: pgmap v188: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:28.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:28 smithi027 ceph-mon[93635]: pgmap v188: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:28.814 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.816+0000 7f498e200700 1 -- 172.21.15.27:0/875282751 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4988111b40 con 0x7f496c06ec00 2024-04-16T05:50:28.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.818+0000 7f497f400700 1 -- 172.21.15.27:0/875282751 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4988111b40 con 0x7f496c06ec00 2024-04-16T05:50:28.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.820+0000 7f498e200700 1 -- 172.21.15.27:0/875282751 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f496c06ec00 msgr2=0x7f496c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:28.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.820+0000 7f498e200700 1 --2- 172.21.15.27:0/875282751 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f496c06ec00 0x7f496c0710b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f497800b970 tx=0x7f4978009e30 comp rx=0 tx=0).stop 2024-04-16T05:50:28.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.820+0000 7f498e200700 1 -- 172.21.15.27:0/875282751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49880fe0e0 msgr2=0x7f4988110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:28.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.820+0000 7f498e200700 1 --2- 172.21.15.27:0/875282751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49880fe0e0 0x7f4988110290 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f497400c810 tx=0x7f4974004470 comp rx=0 tx=0).stop 2024-04-16T05:50:28.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.820+0000 7f498e200700 1 -- 172.21.15.27:0/875282751 shutdown_connections 2024-04-16T05:50:28.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.821+0000 7f498e200700 1 --2- 172.21.15.27:0/875282751 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f496c06ec00 0x7f496c0710b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:28.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.821+0000 7f498e200700 1 --2- 172.21.15.27:0/875282751 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49880fcef0 0x7f498810fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:28.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.821+0000 7f498e200700 1 --2- 172.21.15.27:0/875282751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49880fe0e0 0x7f4988110290 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:28.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.821+0000 7f498e200700 1 -- 172.21.15.27:0/875282751 >> 172.21.15.27:0/875282751 conn(0x7f49880f8530 msgr2=0x7f49880f9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:50:28.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.821+0000 7f498e200700 1 -- 172.21.15.27:0/875282751 shutdown_connections 2024-04-16T05:50:28.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.821+0000 7f498e200700 1 -- 172.21.15.27:0/875282751 wait complete. 2024-04-16T05:50:28.834 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:50:28.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.988+0000 7f6a1e800700 1 -- 172.21.15.27:0/2978884069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a180662f0 msgr2=0x7f6a180ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:28.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.988+0000 7f6a1e800700 1 --2- 172.21.15.27:0/2978884069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a180662f0 0x7f6a180ffa30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6a08004940 tx=0x7f6a08009e30 comp rx=0 tx=0).stop 2024-04-16T05:50:28.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.988+0000 7f6a1e800700 1 -- 172.21.15.27:0/2978884069 shutdown_connections 2024-04-16T05:50:28.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.988+0000 7f6a1e800700 1 --2- 172.21.15.27:0/2978884069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a180fff70 0x7f6a18102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:28.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.988+0000 7f6a1e800700 1 --2- 172.21.15.27:0/2978884069 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a180662f0 0x7f6a180ffa30 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:28.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.988+0000 7f6a1e800700 1 -- 172.21.15.27:0/2978884069 >> 172.21.15.27:0/2978884069 conn(0x7f6a180f74c0 msgr2=0x7f6a180f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:50:28.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.988+0000 7f6a1e800700 1 -- 172.21.15.27:0/2978884069 shutdown_connections 2024-04-16T05:50:28.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.988+0000 7f6a1e800700 1 -- 172.21.15.27:0/2978884069 wait complete. 2024-04-16T05:50:28.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.989+0000 7f6a1e800700 1 Processor -- start 2024-04-16T05:50:28.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.989+0000 7f6a1e800700 1 -- start start 2024-04-16T05:50:28.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.989+0000 7f6a1e800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a180662f0 0x7f6a180fd410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:50:28.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.990+0000 7f6a1e800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a180fff70 0x7f6a180fb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:50:28.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.990+0000 7f6a1e800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a180fbf50 con 0x7f6a180662f0 2024-04-16T05:50:28.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.990+0000 7f6a1e800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a180fc090 con 0x7f6a180fff70 2024-04-16T05:50:28.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.990+0000 7f6a17400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a180662f0 0x7f6a180fd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:50:28.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.990+0000 7f6a17400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a180662f0 0x7f6a180fd410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37320/0 (socket says 172.21.15.27:37320) 2024-04-16T05:50:28.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.990+0000 7f6a17400700 1 -- 172.21.15.27:0/487878115 learned_addr learned my addr 172.21.15.27:0/487878115 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:50:28.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.990+0000 7f6a17400700 1 -- 172.21.15.27:0/487878115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a180fff70 msgr2=0x7f6a180fb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:28.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.990+0000 7f6a16a00700 1 --2- 172.21.15.27:0/487878115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a180fff70 0x7f6a180fb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:50:28.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.990+0000 7f6a17400700 1 --2- 172.21.15.27:0/487878115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a180fff70 0x7f6a180fb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:28.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.990+0000 7f6a17400700 1 -- 172.21.15.27:0/487878115 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a08005040 con 0x7f6a180662f0 2024-04-16T05:50:28.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.990+0000 7f6a16a00700 1 --2- 172.21.15.27:0/487878115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a180fff70 0x7f6a180fb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:50:28.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.991+0000 7f6a17400700 1 --2- 172.21.15.27:0/487878115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a180662f0 0x7f6a180fd410 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6a0800b3b0 tx=0x7f6a08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:50:28.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.991+0000 7f6a0f400700 1 -- 172.21.15.27:0/487878115 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a0800e040 con 0x7f6a180662f0 2024-04-16T05:50:28.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.991+0000 7f6a0f400700 1 -- 172.21.15.27:0/487878115 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a0800bcc0 con 0x7f6a180662f0 2024-04-16T05:50:28.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.991+0000 7f6a1e800700 1 -- 172.21.15.27:0/487878115 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a180fc310 con 0x7f6a180662f0 2024-04-16T05:50:28.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.991+0000 7f6a0f400700 1 -- 172.21.15.27:0/487878115 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a0801c630 con 0x7f6a180662f0 2024-04-16T05:50:28.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.991+0000 7f6a1e800700 1 -- 172.21.15.27:0/487878115 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a18115ac0 con 0x7f6a180662f0 2024-04-16T05:50:28.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.993+0000 7f6a1e800700 1 -- 172.21.15.27:0/487878115 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a1805f5f0 con 0x7f6a180662f0 2024-04-16T05:50:28.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.994+0000 7f6a0f400700 1 -- 172.21.15.27:0/487878115 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6a080139f0 con 0x7f6a180662f0 2024-04-16T05:50:28.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.995+0000 7f6a0f400700 1 --2- 172.21.15.27:0/487878115 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f69fc06ec00 0x7f69fc0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:50:28.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.995+0000 7f6a0f400700 1 -- 172.21.15.27:0/487878115 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f6a0808f5c0 con 0x7f6a180662f0 2024-04-16T05:50:28.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.995+0000 7f6a16a00700 1 --2- 172.21.15.27:0/487878115 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f69fc06ec00 0x7f69fc0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:50:28.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:28.996+0000 7f6a16a00700 1 --2- 172.21.15.27:0/487878115 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f69fc06ec00 0x7f69fc0710b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6a04006fd0 tx=0x7f6a04008040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:50:29.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.005+0000 7f6a0f400700 1 -- 172.21.15.27:0/487878115 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6a08058300 con 0x7f6a180662f0 2024-04-16T05:50:29.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.274+0000 7f6a1e800700 1 -- 172.21.15.27:0/487878115 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a180f8fb0 con 0x7f69fc06ec00 2024-04-16T05:50:29.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.276+0000 7f6a0f400700 1 -- 172.21.15.27:0/487878115 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f6a180f8fb0 con 0x7f69fc06ec00 2024-04-16T05:50:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.279+0000 7f6a1e800700 1 -- 172.21.15.27:0/487878115 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f69fc06ec00 msgr2=0x7f69fc0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.279+0000 7f6a1e800700 1 --2- 172.21.15.27:0/487878115 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f69fc06ec00 0x7f69fc0710b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6a04006fd0 tx=0x7f6a04008040 comp rx=0 tx=0).stop 2024-04-16T05:50:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.279+0000 7f6a1e800700 1 -- 172.21.15.27:0/487878115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a180662f0 msgr2=0x7f6a180fd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.279+0000 7f6a1e800700 1 --2- 172.21.15.27:0/487878115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a180662f0 0x7f6a180fd410 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6a0800b3b0 tx=0x7f6a08009e30 comp rx=0 tx=0).stop 2024-04-16T05:50:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.279+0000 7f6a1e800700 1 -- 172.21.15.27:0/487878115 shutdown_connections 2024-04-16T05:50:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.279+0000 7f6a1e800700 1 --2- 172.21.15.27:0/487878115 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f69fc06ec00 0x7f69fc0710b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.279+0000 7f6a1e800700 1 --2- 172.21.15.27:0/487878115 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a180662f0 0x7f6a180fd410 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.279+0000 7f6a1e800700 1 --2- 172.21.15.27:0/487878115 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a180fff70 0x7f6a180fb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.279+0000 7f6a1e800700 1 -- 172.21.15.27:0/487878115 >> 172.21.15.27:0/487878115 conn(0x7f6a180f74c0 msgr2=0x7f6a180f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:50:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.279+0000 7f6a1e800700 1 -- 172.21.15.27:0/487878115 shutdown_connections 2024-04-16T05:50:29.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.279+0000 7f6a1e800700 1 -- 172.21.15.27:0/487878115 wait complete. 2024-04-16T05:50:29.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.434+0000 7fd568a00700 1 -- 172.21.15.27:0/3967826588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd564071f50 msgr2=0x7fd564070460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:29.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.434+0000 7fd568a00700 1 --2- 172.21.15.27:0/3967826588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd564071f50 0x7fd564070460 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd554009ce0 tx=0x7fd554004d10 comp rx=0 tx=0).stop 2024-04-16T05:50:29.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.434+0000 7fd568a00700 1 -- 172.21.15.27:0/3967826588 shutdown_connections 2024-04-16T05:50:29.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.434+0000 7fd568a00700 1 --2- 172.21.15.27:0/3967826588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5640709a0 0x7fd564070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:29.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.434+0000 7fd568a00700 1 --2- 172.21.15.27:0/3967826588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd564071f50 0x7fd564070460 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:29.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.434+0000 7fd568a00700 1 -- 172.21.15.27:0/3967826588 >> 172.21.15.27:0/3967826588 conn(0x7fd5640f8530 msgr2=0x7fd5640fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:50:29.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.434+0000 7fd568a00700 1 -- 172.21.15.27:0/3967826588 shutdown_connections 2024-04-16T05:50:29.432 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.434+0000 7fd568a00700 1 -- 172.21.15.27:0/3967826588 wait complete. 2024-04-16T05:50:29.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.435+0000 7fd568a00700 1 Processor -- start 2024-04-16T05:50:29.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.435+0000 7fd568a00700 1 -- start start 2024-04-16T05:50:29.433 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.436+0000 7fd568a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5640709a0 0x7fd56410fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:50:29.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.436+0000 7fd568a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd564071f50 0x7fd5641103e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:50:29.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.436+0000 7fd568a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd564111100 con 0x7fd5640709a0 2024-04-16T05:50:29.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.436+0000 7fd568a00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd564111270 con 0x7fd564071f50 2024-04-16T05:50:29.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.436+0000 7fd561e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5640709a0 0x7fd56410fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:50:29.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.436+0000 7fd561e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5640709a0 0x7fd56410fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37340/0 (socket says 172.21.15.27:37340) 2024-04-16T05:50:29.434 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.436+0000 7fd561e00700 1 -- 172.21.15.27:0/3062540397 learned_addr learned my addr 172.21.15.27:0/3062540397 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:50:29.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.436+0000 7fd561e00700 1 -- 172.21.15.27:0/3062540397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd564071f50 msgr2=0x7fd5641103e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:29.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.436+0000 7fd55be00700 1 --2- 172.21.15.27:0/3062540397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd564071f50 0x7fd5641103e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:50:29.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.436+0000 7fd561e00700 1 --2- 172.21.15.27:0/3062540397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd564071f50 0x7fd5641103e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:29.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.436+0000 7fd561e00700 1 -- 172.21.15.27:0/3062540397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd554009940 con 0x7fd5640709a0 2024-04-16T05:50:29.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.436+0000 7fd55be00700 1 --2- 172.21.15.27:0/3062540397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd564071f50 0x7fd5641103e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:50:29.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.437+0000 7fd561e00700 1 --2- 172.21.15.27:0/3062540397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5640709a0 0x7fd56410fea0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd554009910 tx=0x7fd554004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:50:29.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.437+0000 7fd55a000700 1 -- 172.21.15.27:0/3062540397 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd55400e040 con 0x7fd5640709a0 2024-04-16T05:50:29.435 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.437+0000 7fd55a000700 1 -- 172.21.15.27:0/3062540397 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd554008ed0 con 0x7fd5640709a0 2024-04-16T05:50:29.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.437+0000 7fd568a00700 1 -- 172.21.15.27:0/3062540397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd564111490 con 0x7fd5640709a0 2024-04-16T05:50:29.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.437+0000 7fd55a000700 1 -- 172.21.15.27:0/3062540397 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd55401c800 con 0x7fd5640709a0 2024-04-16T05:50:29.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.437+0000 7fd568a00700 1 -- 172.21.15.27:0/3062540397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd564111a50 con 0x7fd5640709a0 2024-04-16T05:50:29.436 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.438+0000 7fd568a00700 1 -- 172.21.15.27:0/3062540397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd56405f5f0 con 0x7fd5640709a0 2024-04-16T05:50:29.437 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.439+0000 7fd55a000700 1 -- 172.21.15.27:0/3062540397 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd554012410 con 0x7fd5640709a0 2024-04-16T05:50:29.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.440+0000 7fd55a000700 1 --2- 172.21.15.27:0/3062540397 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd54806ec10 0x7fd5480710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:50:29.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.440+0000 7fd55a000700 1 -- 172.21.15.27:0/3062540397 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6736+0+0 (secure 0 0 0) 0x7fd5540905f0 con 0x7fd5640709a0 2024-04-16T05:50:29.438 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.441+0000 7fd55be00700 1 --2- 172.21.15.27:0/3062540397 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd54806ec10 0x7fd5480710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:50:29.439 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.441+0000 7fd55be00700 1 --2- 172.21.15.27:0/3062540397 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd54806ec10 0x7fd5480710c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd54c006fd0 tx=0x7fd54c008040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:50:29.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.446+0000 7fd55a000700 1 -- 172.21.15.27:0/3062540397 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd5540592b0 con 0x7fd5640709a0 2024-04-16T05:50:29.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.711+0000 7fd568a00700 1 -- 172.21.15.27:0/3062540397 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd564111d80 con 0x7fd54806ec10 2024-04-16T05:50:29.709 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:50:29.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.722+0000 7fd55a000700 1 -- 172.21.15.27:0/3062540397 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fd564111d80 con 0x7fd54806ec10 2024-04-16T05:50:29.720 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:50:29.720 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (6m) 106s ago 17m 13.0M - 0.25.0 c8568f914cd2 4ed63a8a6b33 2024-04-16T05:50:29.720 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (3m) 106s ago 17m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 4aa1f0006685 2024-04-16T05:50:29.720 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (3m) 24s ago 16m 6656k - 19.0.0-2884-g218a6914 a4e4811815e4 1c1912507d56 2024-04-16T05:50:29.720 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (6m) 106s ago 17m 75.4M - 9.4.7 954c08fa6188 f7ee2a1afd12 2024-04-16T05:50:29.720 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (10m) 106s ago 10m 23.7M - 17.2.7-847-g45317f95 b333581ccdf5 0824e2621be5 2024-04-16T05:50:29.720 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 starting - - - - 2024-04-16T05:50:29.720 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (10m) 24s ago 10m 14.4M - 17.2.7-847-g45317f95 b333581ccdf5 f5e36815ddfa 2024-04-16T05:50:29.720 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (10m) 24s ago 10m 14.3M - 17.2.7-847-g45317f95 b333581ccdf5 755a73c65a7f 2024-04-16T05:50:29.721 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:8443,9283,8765 running (6m) 106s ago 19m 502M - 19.0.0-2884-g218a6914 a4e4811815e4 b69a0b08e12b 2024-04-16T05:50:29.721 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283,8765 running (5m) 24s ago 16m 453M - 19.0.0-2884-g218a6914 a4e4811815e4 32e27bcd7e42 2024-04-16T05:50:29.721 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (5m) 106s ago 19m 45.4M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 299ca337e4ca 2024-04-16T05:50:29.721 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (4m) 24s ago 16m 34.4M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5db5f3037ed0 2024-04-16T05:50:29.721 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (6m) 106s ago 17m 8086k - 1.5.0 0da6a335fe13 5e29e29868b0 2024-04-16T05:50:29.721 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (6m) 24s ago 16m 7910k - 1.5.0 0da6a335fe13 a360922bc621 2024-04-16T05:50:29.721 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (2m) 106s ago 15m 42.5M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 cef03b67bc5a 2024-04-16T05:50:29.721 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (2m) 106s ago 14m 42.2M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3a6ab1fc6e79 2024-04-16T05:50:29.721 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (114s) 106s ago 13m 11.1M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3524731d7277 2024-04-16T05:50:29.721 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (83s) 24s ago 13m 42.7M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 897d3579f929 2024-04-16T05:50:29.721 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (56s) 24s ago 12m 41.2M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 fa97758ff7c5 2024-04-16T05:50:29.721 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (29s) 24s ago 12m 11.2M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 fd12260a820d 2024-04-16T05:50:29.721 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (5m) 106s ago 16m 57.0M - 2.43.0 a07b618ecd1d e70f55ba5ec0 2024-04-16T05:50:29.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.725+0000 7fd568a00700 1 -- 172.21.15.27:0/3062540397 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd54806ec10 msgr2=0x7fd5480710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:29.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.725+0000 7fd568a00700 1 --2- 172.21.15.27:0/3062540397 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd54806ec10 0x7fd5480710c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd54c006fd0 tx=0x7fd54c008040 comp rx=0 tx=0).stop 2024-04-16T05:50:29.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.725+0000 7fd568a00700 1 -- 172.21.15.27:0/3062540397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5640709a0 msgr2=0x7fd56410fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:29.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.725+0000 7fd568a00700 1 --2- 172.21.15.27:0/3062540397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5640709a0 0x7fd56410fea0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd554009910 tx=0x7fd554004d10 comp rx=0 tx=0).stop 2024-04-16T05:50:29.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.725+0000 7fd568a00700 1 -- 172.21.15.27:0/3062540397 shutdown_connections 2024-04-16T05:50:29.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.725+0000 7fd568a00700 1 --2- 172.21.15.27:0/3062540397 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd54806ec10 0x7fd5480710c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:29.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.725+0000 7fd568a00700 1 --2- 172.21.15.27:0/3062540397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd5640709a0 0x7fd56410fea0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:29.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.725+0000 7fd568a00700 1 --2- 172.21.15.27:0/3062540397 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd564071f50 0x7fd5641103e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:29.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.725+0000 7fd568a00700 1 -- 172.21.15.27:0/3062540397 >> 172.21.15.27:0/3062540397 conn(0x7fd5640f8530 msgr2=0x7fd564102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:50:29.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.725+0000 7fd568a00700 1 -- 172.21.15.27:0/3062540397 shutdown_connections 2024-04-16T05:50:29.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.725+0000 7fd568a00700 1 -- 172.21.15.27:0/3062540397 wait complete. 2024-04-16T05:50:29.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:50:29.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.883+0000 7f1f0fe00700 1 -- 172.21.15.27:0/3909125844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f100fef00 msgr2=0x7f1f100ff310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:29.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.883+0000 7f1f0fe00700 1 --2- 172.21.15.27:0/3909125844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f100fef00 0x7f1f100ff310 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1f00004940 tx=0x7f1f00009e30 comp rx=0 tx=0).stop 2024-04-16T05:50:29.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.884+0000 7f1f0fe00700 1 -- 172.21.15.27:0/3909125844 shutdown_connections 2024-04-16T05:50:29.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.884+0000 7f1f0fe00700 1 --2- 172.21.15.27:0/3909125844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1f10100070 0x7f1f101004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:29.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.884+0000 7f1f0fe00700 1 --2- 172.21.15.27:0/3909125844 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f100fef00 0x7f1f100ff310 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:29.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.884+0000 7f1f0fe00700 1 -- 172.21.15.27:0/3909125844 >> 172.21.15.27:0/3909125844 conn(0x7f1f100fa790 msgr2=0x7f1f100fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:50:29.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.884+0000 7f1f0fe00700 1 -- 172.21.15.27:0/3909125844 shutdown_connections 2024-04-16T05:50:29.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.884+0000 7f1f0fe00700 1 -- 172.21.15.27:0/3909125844 wait complete. 2024-04-16T05:50:29.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.885+0000 7f1f0fe00700 1 Processor -- start 2024-04-16T05:50:29.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.885+0000 7f1f0fe00700 1 -- start start 2024-04-16T05:50:29.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.885+0000 7f1f0fe00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1f100fef00 0x7f1f1010fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:50:29.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.885+0000 7f1f0fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f10100070 0x7f1f1010ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:50:29.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.885+0000 7f1f0fe00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f101105a0 con 0x7f1f10100070 2024-04-16T05:50:29.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.885+0000 7f1f0fe00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f10110df0 con 0x7f1f100fef00 2024-04-16T05:50:29.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.885+0000 7f1f0e000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f10100070 0x7f1f1010ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:50:29.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.885+0000 7f1f0e000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f10100070 0x7f1f1010ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37350/0 (socket says 172.21.15.27:37350) 2024-04-16T05:50:29.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.885+0000 7f1f0e000700 1 -- 172.21.15.27:0/1218297423 learned_addr learned my addr 172.21.15.27:0/1218297423 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:50:29.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.886+0000 7f1f0ea00700 1 --2- 172.21.15.27:0/1218297423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1f100fef00 0x7f1f1010fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:50:29.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.886+0000 7f1f0e000700 1 -- 172.21.15.27:0/1218297423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1f100fef00 msgr2=0x7f1f1010fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:29.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.886+0000 7f1f0e000700 1 --2- 172.21.15.27:0/1218297423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1f100fef00 0x7f1f1010fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:29.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.886+0000 7f1f0e000700 1 -- 172.21.15.27:0/1218297423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f00005040 con 0x7f1f10100070 2024-04-16T05:50:29.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.886+0000 7f1f0ea00700 1 --2- 172.21.15.27:0/1218297423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1f100fef00 0x7f1f1010fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:50:29.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.886+0000 7f1f0e000700 1 --2- 172.21.15.27:0/1218297423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f10100070 0x7f1f1010ffa0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1efc002a50 tx=0x7f1efc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:50:29.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.887+0000 7f1f06a00700 1 -- 172.21.15.27:0/1218297423 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1efc007cc0 con 0x7f1f10100070 2024-04-16T05:50:29.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.887+0000 7f1f06a00700 1 -- 172.21.15.27:0/1218297423 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1efc00f040 con 0x7f1f10100070 2024-04-16T05:50:29.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.887+0000 7f1f06a00700 1 -- 172.21.15.27:0/1218297423 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1efc009640 con 0x7f1f10100070 2024-04-16T05:50:29.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.887+0000 7f1f0fe00700 1 -- 172.21.15.27:0/1218297423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f101110d0 con 0x7f1f10100070 2024-04-16T05:50:29.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.887+0000 7f1f0fe00700 1 -- 172.21.15.27:0/1218297423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f10111610 con 0x7f1f10100070 2024-04-16T05:50:29.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.888+0000 7f1f0fe00700 1 -- 172.21.15.27:0/1218297423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f100fe290 con 0x7f1f10100070 2024-04-16T05:50:29.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.889+0000 7f1f06a00700 1 -- 172.21.15.27:0/1218297423 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1efc0097a0 con 0x7f1f10100070 2024-04-16T05:50:29.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.890+0000 7f1f06a00700 1 --2- 172.21.15.27:0/1218297423 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1ef406ec00 0x7f1ef40710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:50:29.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.890+0000 7f1f06a00700 1 -- 172.21.15.27:0/1218297423 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f1efc08e720 con 0x7f1f10100070 2024-04-16T05:50:29.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.890+0000 7f1f0ea00700 1 --2- 172.21.15.27:0/1218297423 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1ef406ec00 0x7f1ef40710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:50:29.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.891+0000 7f1f0ea00700 1 --2- 172.21.15.27:0/1218297423 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1ef406ec00 0x7f1ef40710b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1f0000b970 tx=0x7f1f00009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:50:29.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:29.895+0000 7f1f06a00700 1 -- 172.21.15.27:0/1218297423 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1efc05af10 con 0x7f1f10100070 2024-04-16T05:50:30.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.253+0000 7f1f0fe00700 1 -- 172.21.15.27:0/1218297423 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1f101119d0 con 0x7f1f10100070 2024-04-16T05:50:30.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.254+0000 7f1f06a00700 1 -- 172.21.15.27:0/1218297423 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f1efc05aaa0 con 0x7f1f10100070 2024-04-16T05:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:50:30.253 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T05:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 3, 2024-04-16T05:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 1 2024-04-16T05:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 3, 2024-04-16T05:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 11 2024-04-16T05:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:50:30.254 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:50:30.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.258+0000 7f1f0fe00700 1 -- 172.21.15.27:0/1218297423 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1ef406ec00 msgr2=0x7f1ef40710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:30.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.258+0000 7f1f0fe00700 1 --2- 172.21.15.27:0/1218297423 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1ef406ec00 0x7f1ef40710b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1f0000b970 tx=0x7f1f00009e30 comp rx=0 tx=0).stop 2024-04-16T05:50:30.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.258+0000 7f1f0fe00700 1 -- 172.21.15.27:0/1218297423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f10100070 msgr2=0x7f1f1010ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:30.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.258+0000 7f1f0fe00700 1 --2- 172.21.15.27:0/1218297423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f10100070 0x7f1f1010ffa0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1efc002a50 tx=0x7f1efc0046b0 comp rx=0 tx=0).stop 2024-04-16T05:50:30.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.258+0000 7f1f0fe00700 1 -- 172.21.15.27:0/1218297423 shutdown_connections 2024-04-16T05:50:30.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.258+0000 7f1f0fe00700 1 --2- 172.21.15.27:0/1218297423 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1ef406ec00 0x7f1ef40710b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:30.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.258+0000 7f1f0fe00700 1 --2- 172.21.15.27:0/1218297423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1f100fef00 0x7f1f1010fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:30.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.258+0000 7f1f0fe00700 1 --2- 172.21.15.27:0/1218297423 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f10100070 0x7f1f1010ffa0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:30.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.258+0000 7f1f0fe00700 1 -- 172.21.15.27:0/1218297423 >> 172.21.15.27:0/1218297423 conn(0x7f1f100fa790 msgr2=0x7f1f100fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:50:30.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.258+0000 7f1f0fe00700 1 -- 172.21.15.27:0/1218297423 shutdown_connections 2024-04-16T05:50:30.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.258+0000 7f1f0fe00700 1 -- 172.21.15.27:0/1218297423 wait complete. 2024-04-16T05:50:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.415+0000 7fd7c7a00700 1 -- 172.21.15.27:0/94382132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7c00fdd60 msgr2=0x7fd7c01001e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.415+0000 7fd7c7a00700 1 --2- 172.21.15.27:0/94382132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7c00fdd60 0x7fd7c01001e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac004940 tx=0x7fd7ac009e30 comp rx=0 tx=0).stop 2024-04-16T05:50:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.415+0000 7fd7c7a00700 1 -- 172.21.15.27:0/94382132 shutdown_connections 2024-04-16T05:50:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.415+0000 7fd7c7a00700 1 --2- 172.21.15.27:0/94382132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7c00fdd60 0x7fd7c01001e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.415+0000 7fd7c7a00700 1 --2- 172.21.15.27:0/94382132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7c00662f0 0x7fd7c00fd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.415+0000 7fd7c7a00700 1 -- 172.21.15.27:0/94382132 >> 172.21.15.27:0/94382132 conn(0x7fd7c00f74c0 msgr2=0x7fd7c00f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:50:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.415+0000 7fd7c7a00700 1 -- 172.21.15.27:0/94382132 shutdown_connections 2024-04-16T05:50:30.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.415+0000 7fd7c7a00700 1 -- 172.21.15.27:0/94382132 wait complete. 2024-04-16T05:50:30.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.416+0000 7fd7c7a00700 1 Processor -- start 2024-04-16T05:50:30.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.416+0000 7fd7c7a00700 1 -- start start 2024-04-16T05:50:30.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.416+0000 7fd7c7a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7c00662f0 0x7fd7c0114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:50:30.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.416+0000 7fd7c7a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7c00fdd60 0x7fd7c01145a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:50:30.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.416+0000 7fd7c7a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c0114ba0 con 0x7fd7c00fdd60 2024-04-16T05:50:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.416+0000 7fd7c7a00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c01153f0 con 0x7fd7c00662f0 2024-04-16T05:50:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.417+0000 7fd7bfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7c00fdd60 0x7fd7c01145a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:50:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.417+0000 7fd7bfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7c00fdd60 0x7fd7c01145a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37358/0 (socket says 172.21.15.27:37358) 2024-04-16T05:50:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.417+0000 7fd7bfe00700 1 -- 172.21.15.27:0/1510241358 learned_addr learned my addr 172.21.15.27:0/1510241358 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:50:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.417+0000 7fd7c5000700 1 --2- 172.21.15.27:0/1510241358 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7c00662f0 0x7fd7c0114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:50:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.417+0000 7fd7bfe00700 1 -- 172.21.15.27:0/1510241358 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7c00662f0 msgr2=0x7fd7c0114060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.417+0000 7fd7bfe00700 1 --2- 172.21.15.27:0/1510241358 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7c00662f0 0x7fd7c0114060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.417+0000 7fd7bfe00700 1 -- 172.21.15.27:0/1510241358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ac005040 con 0x7fd7c00fdd60 2024-04-16T05:50:30.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.417+0000 7fd7bfe00700 1 --2- 172.21.15.27:0/1510241358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7c00fdd60 0x7fd7c01145a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac00b3b0 tx=0x7fd7ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:50:30.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.418+0000 7fd7bcc00700 1 -- 172.21.15.27:0/1510241358 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7ac00e040 con 0x7fd7c00fdd60 2024-04-16T05:50:30.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.418+0000 7fd7bcc00700 1 -- 172.21.15.27:0/1510241358 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd7ac004290 con 0x7fd7c00fdd60 2024-04-16T05:50:30.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.418+0000 7fd7c7a00700 1 -- 172.21.15.27:0/1510241358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c0115670 con 0x7fd7c00fdd60 2024-04-16T05:50:30.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.418+0000 7fd7bcc00700 1 -- 172.21.15.27:0/1510241358 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7ac01c630 con 0x7fd7c00fdd60 2024-04-16T05:50:30.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.418+0000 7fd7c7a00700 1 -- 172.21.15.27:0/1510241358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c0115bb0 con 0x7fd7c00fdd60 2024-04-16T05:50:30.417 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.419+0000 7fd7c7a00700 1 -- 172.21.15.27:0/1510241358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c005f5f0 con 0x7fd7c00fdd60 2024-04-16T05:50:30.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.421+0000 7fd7bcc00700 1 -- 172.21.15.27:0/1510241358 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd7ac00bbf0 con 0x7fd7c00fdd60 2024-04-16T05:50:30.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.422+0000 7fd7bcc00700 1 --2- 172.21.15.27:0/1510241358 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd7a806ec10 0x7fd7a80710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:50:30.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.422+0000 7fd7bcc00700 1 -- 172.21.15.27:0/1510241358 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6736+0+0 (secure 0 0 0) 0x7fd7ac08fca0 con 0x7fd7c00fdd60 2024-04-16T05:50:30.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.422+0000 7fd7c5000700 1 --2- 172.21.15.27:0/1510241358 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd7a806ec10 0x7fd7a80710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:50:30.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.423+0000 7fd7c5000700 1 --2- 172.21.15.27:0/1510241358 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd7a806ec10 0x7fd7a80710c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0066030 tx=0x7fd7b4009ee0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:50:30.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.427+0000 7fd7bcc00700 1 -- 172.21.15.27:0/1510241358 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd7ac094070 con 0x7fd7c00fdd60 2024-04-16T05:50:30.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:30 smithi027 ceph-mon[93635]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:50:30.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:30 smithi027 ceph-mon[93635]: pgmap v189: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:30.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:30 smithi027 ceph-mon[93635]: from='client.34408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:50:30.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:30 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1218297423' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:50:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.716+0000 7fd7c7a00700 1 -- 172.21.15.27:0/1510241358 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd7c0115ec0 con 0x7fd7c00fdd60 2024-04-16T05:50:30.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.716+0000 7fd7bcc00700 1 -- 172.21.15.27:0/1510241358 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 18 v18) v1 ==== 76+0+1881 (secure 0 0 0) 0x7fd7ac05c5d0 con 0x7fd7c00fdd60 2024-04-16T05:50:30.715 INFO:teuthology.orchestra.run.smithi027.stdout:e18 2024-04-16T05:50:30.715 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:50:30.715 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:50:30.715 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:50:30.715 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:50:30.715 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:50:30.715 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:50:30.715 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 15 2024-04-16T05:50:30.715 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:50:30.715 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:50:30.715 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:40:32.701649+0000 2024-04-16T05:50:30.715 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:50:30.715 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:50:30.715 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:50:30.716 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:50:30.716 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:50:30.716 INFO:teuthology.orchestra.run.smithi027.stdout:max_xattr_size 65536 2024-04-16T05:50:30.716 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:50:30.716 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:50:30.716 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 46 2024-04-16T05:50:30.716 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:50:30.716 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:50:30.716 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:50:30.716 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=14462} 2024-04-16T05:50:30.716 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:50:30.716 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:50:30.716 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:50:30.716 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:50:30.716 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:50:30.716 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:50:30.717 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:50:30.717 INFO:teuthology.orchestra.run.smithi027.stdout:bal_rank_mask -1 2024-04-16T05:50:30.717 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:50:30.717 INFO:teuthology.orchestra.run.smithi027.stdout:qdb_cluster leader: 0 members: 2024-04-16T05:50:30.717 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{0:14462} state up:active seq 5 join_fscid=1 addr [v2:172.21.15.27:6828/4115272713,v1:172.21.15.27:6829/4115272713] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:50:30.717 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:50:30.717 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:50:30.717 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:50:30.717 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:50:30.717 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:24263} state up:standby seq 4 join_fscid=1 addr [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:50:30.717 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:24273} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:50:30.717 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{-1:34400} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2831468256,v1:172.21.15.27:6827/2831468256] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:50:30.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.721+0000 7fd7c7a00700 1 -- 172.21.15.27:0/1510241358 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd7a806ec10 msgr2=0x7fd7a80710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:30.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.721+0000 7fd7c7a00700 1 --2- 172.21.15.27:0/1510241358 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd7a806ec10 0x7fd7a80710c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0066030 tx=0x7fd7b4009ee0 comp rx=0 tx=0).stop 2024-04-16T05:50:30.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.721+0000 7fd7c7a00700 1 -- 172.21.15.27:0/1510241358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7c00fdd60 msgr2=0x7fd7c01145a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:30.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.721+0000 7fd7c7a00700 1 --2- 172.21.15.27:0/1510241358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7c00fdd60 0x7fd7c01145a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac00b3b0 tx=0x7fd7ac009e30 comp rx=0 tx=0).stop 2024-04-16T05:50:30.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.721+0000 7fd7c7a00700 1 -- 172.21.15.27:0/1510241358 shutdown_connections 2024-04-16T05:50:30.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.721+0000 7fd7c7a00700 1 --2- 172.21.15.27:0/1510241358 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd7a806ec10 0x7fd7a80710c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:30.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.721+0000 7fd7c7a00700 1 --2- 172.21.15.27:0/1510241358 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7c00662f0 0x7fd7c0114060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:30.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.721+0000 7fd7c7a00700 1 --2- 172.21.15.27:0/1510241358 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd7c00fdd60 0x7fd7c01145a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:30.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.721+0000 7fd7c7a00700 1 -- 172.21.15.27:0/1510241358 >> 172.21.15.27:0/1510241358 conn(0x7fd7c00f74c0 msgr2=0x7fd7c00f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:50:30.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.721+0000 7fd7c7a00700 1 -- 172.21.15.27:0/1510241358 shutdown_connections 2024-04-16T05:50:30.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.721+0000 7fd7c7a00700 1 -- 172.21.15.27:0/1510241358 wait complete. 2024-04-16T05:50:30.720 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 18 2024-04-16T05:50:30.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:30 smithi122 ceph-mon[80930]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:50:30.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:30 smithi122 ceph-mon[80930]: pgmap v189: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:30.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:30 smithi122 ceph-mon[80930]: from='client.34408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:50:30.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:30 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1218297423' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:50:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.879+0000 7fdd79800700 1 -- 172.21.15.27:0/2056147041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740fce40 msgr2=0x7fdd740fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.879+0000 7fdd79800700 1 --2- 172.21.15.27:0/2056147041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740fce40 0x7fdd740fd250 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdd64004940 tx=0x7fdd64009e30 comp rx=0 tx=0).stop 2024-04-16T05:50:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.879+0000 7fdd79800700 1 -- 172.21.15.27:0/2056147041 shutdown_connections 2024-04-16T05:50:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.879+0000 7fdd79800700 1 --2- 172.21.15.27:0/2056147041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd740fe040 0x7fdd740fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.879+0000 7fdd79800700 1 --2- 172.21.15.27:0/2056147041 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740fce40 0x7fdd740fd250 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.879+0000 7fdd79800700 1 -- 172.21.15.27:0/2056147041 >> 172.21.15.27:0/2056147041 conn(0x7fdd740f8530 msgr2=0x7fdd740fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:50:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.879+0000 7fdd79800700 1 -- 172.21.15.27:0/2056147041 shutdown_connections 2024-04-16T05:50:30.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.879+0000 7fdd79800700 1 -- 172.21.15.27:0/2056147041 wait complete. 2024-04-16T05:50:30.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.880+0000 7fdd79800700 1 Processor -- start 2024-04-16T05:50:30.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.880+0000 7fdd79800700 1 -- start start 2024-04-16T05:50:30.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.881+0000 7fdd79800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd740fce40 0x7fdd7410fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:50:30.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.881+0000 7fdd79800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740fe040 0x7fdd74110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:50:30.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.881+0000 7fdd79800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd74110850 con 0x7fdd740fe040 2024-04-16T05:50:30.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.881+0000 7fdd79800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd74110f90 con 0x7fdd740fce40 2024-04-16T05:50:30.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.881+0000 7fdd72000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740fe040 0x7fdd74110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:50:30.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.881+0000 7fdd72000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740fe040 0x7fdd74110250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37378/0 (socket says 172.21.15.27:37378) 2024-04-16T05:50:30.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.881+0000 7fdd72000700 1 -- 172.21.15.27:0/131188416 learned_addr learned my addr 172.21.15.27:0/131188416 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:50:30.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.881+0000 7fdd72a00700 1 --2- 172.21.15.27:0/131188416 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd740fce40 0x7fdd7410fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:50:30.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.882+0000 7fdd72000700 1 -- 172.21.15.27:0/131188416 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd740fce40 msgr2=0x7fdd7410fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:30.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.882+0000 7fdd72000700 1 --2- 172.21.15.27:0/131188416 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd740fce40 0x7fdd7410fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:30.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.882+0000 7fdd72000700 1 -- 172.21.15.27:0/131188416 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd64005040 con 0x7fdd740fe040 2024-04-16T05:50:30.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.882+0000 7fdd72000700 1 --2- 172.21.15.27:0/131188416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740fe040 0x7fdd74110250 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000c8c0 tx=0x7fdd600044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:50:30.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.882+0000 7fdd6aa00700 1 -- 172.21.15.27:0/131188416 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd6000a980 con 0x7fdd740fe040 2024-04-16T05:50:30.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.882+0000 7fdd79800700 1 -- 172.21.15.27:0/131188416 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd74111270 con 0x7fdd740fe040 2024-04-16T05:50:30.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.882+0000 7fdd6aa00700 1 -- 172.21.15.27:0/131188416 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd6000ce60 con 0x7fdd740fe040 2024-04-16T05:50:30.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.882+0000 7fdd6aa00700 1 -- 172.21.15.27:0/131188416 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd600115e0 con 0x7fdd740fe040 2024-04-16T05:50:30.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.882+0000 7fdd79800700 1 -- 172.21.15.27:0/131188416 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd74111830 con 0x7fdd740fe040 2024-04-16T05:50:30.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.884+0000 7fdd79800700 1 -- 172.21.15.27:0/131188416 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd741019d0 con 0x7fdd740fe040 2024-04-16T05:50:30.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.885+0000 7fdd6aa00700 1 -- 172.21.15.27:0/131188416 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdd60012400 con 0x7fdd740fe040 2024-04-16T05:50:30.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.886+0000 7fdd6aa00700 1 --2- 172.21.15.27:0/131188416 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fdd5806ec00 0x7fdd580710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:50:30.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.886+0000 7fdd6aa00700 1 -- 172.21.15.27:0/131188416 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6736+0+0 (secure 0 0 0) 0x7fdd6008eaf0 con 0x7fdd740fe040 2024-04-16T05:50:30.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.886+0000 7fdd72a00700 1 --2- 172.21.15.27:0/131188416 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fdd5806ec00 0x7fdd580710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:50:30.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.887+0000 7fdd72a00700 1 --2- 172.21.15.27:0/131188416 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fdd5806ec00 0x7fdd580710b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdd6400b970 tx=0x7fdd64009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:50:30.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:30.891+0000 7fdd6aa00700 1 -- 172.21.15.27:0/131188416 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fdd6005b2e0 con 0x7fdd740fe040 2024-04-16T05:50:31.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.159+0000 7fdd79800700 1 -- 172.21.15.27:0/131188416 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd74111ab0 con 0x7fdd5806ec00 2024-04-16T05:50:31.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.161+0000 7fdd6aa00700 1 -- 172.21.15.27:0/131188416 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fdd74111ab0 con 0x7fdd5806ec00 2024-04-16T05:50:31.159 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:50:31.159 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T05:50:31.159 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:50:31.159 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:50:31.159 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [ 2024-04-16T05:50:31.159 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr", 2024-04-16T05:50:31.159 INFO:teuthology.orchestra.run.smithi027.stdout: "mon", 2024-04-16T05:50:31.159 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-16T05:50:31.160 INFO:teuthology.orchestra.run.smithi027.stdout: "osd" 2024-04-16T05:50:31.160 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-16T05:50:31.160 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "12/21 daemons upgraded", 2024-04-16T05:50:31.160 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T05:50:31.160 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:50:31.160 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:50:31.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.164+0000 7fdd79800700 1 -- 172.21.15.27:0/131188416 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fdd5806ec00 msgr2=0x7fdd580710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:31.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.164+0000 7fdd79800700 1 --2- 172.21.15.27:0/131188416 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fdd5806ec00 0x7fdd580710b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdd6400b970 tx=0x7fdd64009e30 comp rx=0 tx=0).stop 2024-04-16T05:50:31.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.164+0000 7fdd79800700 1 -- 172.21.15.27:0/131188416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740fe040 msgr2=0x7fdd74110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.164+0000 7fdd79800700 1 --2- 172.21.15.27:0/131188416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740fe040 0x7fdd74110250 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000c8c0 tx=0x7fdd600044c0 comp rx=0 tx=0).stop 2024-04-16T05:50:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.164+0000 7fdd79800700 1 -- 172.21.15.27:0/131188416 shutdown_connections 2024-04-16T05:50:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.164+0000 7fdd79800700 1 --2- 172.21.15.27:0/131188416 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fdd5806ec00 0x7fdd580710b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.164+0000 7fdd79800700 1 --2- 172.21.15.27:0/131188416 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdd740fce40 0x7fdd7410fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.164+0000 7fdd79800700 1 --2- 172.21.15.27:0/131188416 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd740fe040 0x7fdd74110250 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.164+0000 7fdd79800700 1 -- 172.21.15.27:0/131188416 >> 172.21.15.27:0/131188416 conn(0x7fdd740f8530 msgr2=0x7fdd740f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:50:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.164+0000 7fdd79800700 1 -- 172.21.15.27:0/131188416 shutdown_connections 2024-04-16T05:50:31.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.164+0000 7fdd79800700 1 -- 172.21.15.27:0/131188416 wait complete. 2024-04-16T05:50:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.319+0000 7f06d4e00700 1 -- 172.21.15.27:0/53705201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d0073170 msgr2=0x7f06d0073b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.319+0000 7f06d4e00700 1 --2- 172.21.15.27:0/53705201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d0073170 0x7f06d0073b80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f06bc004940 tx=0x7f06bc009e30 comp rx=0 tx=0).stop 2024-04-16T05:50:31.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.319+0000 7f06d4e00700 1 -- 172.21.15.27:0/53705201 shutdown_connections 2024-04-16T05:50:31.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.319+0000 7f06d4e00700 1 --2- 172.21.15.27:0/53705201 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d0073170 0x7f06d0073b80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:31.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.319+0000 7f06d4e00700 1 --2- 172.21.15.27:0/53705201 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06d0072820 0x7f06d0072c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:31.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.319+0000 7f06d4e00700 1 -- 172.21.15.27:0/53705201 >> 172.21.15.27:0/53705201 conn(0x7f06d00fa950 msgr2=0x7f06d00fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:50:31.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.319+0000 7f06d4e00700 1 -- 172.21.15.27:0/53705201 shutdown_connections 2024-04-16T05:50:31.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.319+0000 7f06d4e00700 1 -- 172.21.15.27:0/53705201 wait complete. 2024-04-16T05:50:31.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.321+0000 7f06d4e00700 1 Processor -- start 2024-04-16T05:50:31.319 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.321+0000 7f06d4e00700 1 -- start start 2024-04-16T05:50:31.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.321+0000 7f06d4e00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06d0072820 0x7f06d01140a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:50:31.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.321+0000 7f06d4e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d01145e0 0x7f06d0115430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:50:31.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.321+0000 7f06d4e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d0114aa0 con 0x7f06d01145e0 2024-04-16T05:50:31.320 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.321+0000 7f06d4e00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d0114be0 con 0x7f06d0072820 2024-04-16T05:50:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.322+0000 7f06cd400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d01145e0 0x7f06d0115430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:50:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.322+0000 7f06cd400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d01145e0 0x7f06d0115430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37396/0 (socket says 172.21.15.27:37396) 2024-04-16T05:50:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.322+0000 7f06cd400700 1 -- 172.21.15.27:0/4090325259 learned_addr learned my addr 172.21.15.27:0/4090325259 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:50:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.322+0000 7f06cde00700 1 --2- 172.21.15.27:0/4090325259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06d0072820 0x7f06d01140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:50:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.322+0000 7f06cd400700 1 -- 172.21.15.27:0/4090325259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06d0072820 msgr2=0x7f06d01140a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.322+0000 7f06cd400700 1 --2- 172.21.15.27:0/4090325259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06d0072820 0x7f06d01140a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.322+0000 7f06cd400700 1 -- 172.21.15.27:0/4090325259 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06bc005040 con 0x7f06d01145e0 2024-04-16T05:50:31.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.323+0000 7f06cd400700 1 --2- 172.21.15.27:0/4090325259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d01145e0 0x7f06d0115430 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f06bc009150 tx=0x7f06bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:50:31.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.323+0000 7f06c6000700 1 -- 172.21.15.27:0/4090325259 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06bc00e040 con 0x7f06d01145e0 2024-04-16T05:50:31.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.323+0000 7f06c6000700 1 -- 172.21.15.27:0/4090325259 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f06bc00bcc0 con 0x7f06d01145e0 2024-04-16T05:50:31.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.323+0000 7f06d4e00700 1 -- 172.21.15.27:0/4090325259 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d0115a30 con 0x7f06d01145e0 2024-04-16T05:50:31.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.323+0000 7f06c6000700 1 -- 172.21.15.27:0/4090325259 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06bc01c630 con 0x7f06d01145e0 2024-04-16T05:50:31.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.323+0000 7f06d4e00700 1 -- 172.21.15.27:0/4090325259 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d0115fc0 con 0x7f06d01145e0 2024-04-16T05:50:31.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.324+0000 7f06d4e00700 1 -- 172.21.15.27:0/4090325259 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d005f5f0 con 0x7f06d01145e0 2024-04-16T05:50:31.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.326+0000 7f06c6000700 1 -- 172.21.15.27:0/4090325259 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f06bc013420 con 0x7f06d01145e0 2024-04-16T05:50:31.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.327+0000 7f06c6000700 1 --2- 172.21.15.27:0/4090325259 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f06b406ec00 0x7f06b40710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:50:31.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.327+0000 7f06c6000700 1 -- 172.21.15.27:0/4090325259 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6736+0+0 (secure 0 0 0) 0x7f06bc08f5e0 con 0x7f06d01145e0 2024-04-16T05:50:31.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.327+0000 7f06cde00700 1 --2- 172.21.15.27:0/4090325259 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f06b406ec00 0x7f06b40710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:50:31.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.328+0000 7f06cde00700 1 --2- 172.21.15.27:0/4090325259 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f06b406ec00 0x7f06b40710b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f06d0072560 tx=0x7f06c0009ee0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:50:31.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.332+0000 7f06c6000700 1 -- 172.21.15.27:0/4090325259 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f06bc05be50 con 0x7f06d01145e0 2024-04-16T05:50:31.664 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.666+0000 7f06d4e00700 1 -- 172.21.15.27:0/4090325259 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f06d004bed0 con 0x7f06d01145e0 2024-04-16T05:50:31.665 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.667+0000 7f06c6000700 1 -- 172.21.15.27:0/4090325259 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f06bc05b9e0 con 0x7f06d01145e0 2024-04-16T05:50:31.665 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:50:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.670+0000 7f06d4e00700 1 -- 172.21.15.27:0/4090325259 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f06b406ec00 msgr2=0x7f06b40710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.670+0000 7f06d4e00700 1 --2- 172.21.15.27:0/4090325259 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f06b406ec00 0x7f06b40710b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f06d0072560 tx=0x7f06c0009ee0 comp rx=0 tx=0).stop 2024-04-16T05:50:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.670+0000 7f06d4e00700 1 -- 172.21.15.27:0/4090325259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d01145e0 msgr2=0x7f06d0115430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:50:31.668 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.670+0000 7f06d4e00700 1 --2- 172.21.15.27:0/4090325259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d01145e0 0x7f06d0115430 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f06bc009150 tx=0x7f06bc009e30 comp rx=0 tx=0).stop 2024-04-16T05:50:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.670+0000 7f06d4e00700 1 -- 172.21.15.27:0/4090325259 shutdown_connections 2024-04-16T05:50:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.670+0000 7f06d4e00700 1 --2- 172.21.15.27:0/4090325259 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f06b406ec00 0x7f06b40710b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.670+0000 7f06d4e00700 1 --2- 172.21.15.27:0/4090325259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06d0072820 0x7f06d01140a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.670+0000 7f06d4e00700 1 --2- 172.21.15.27:0/4090325259 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f06d01145e0 0x7f06d0115430 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:50:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.670+0000 7f06d4e00700 1 -- 172.21.15.27:0/4090325259 >> 172.21.15.27:0/4090325259 conn(0x7f06d00fa950 msgr2=0x7f06d00fd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:50:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.671+0000 7f06d4e00700 1 -- 172.21.15.27:0/4090325259 shutdown_connections 2024-04-16T05:50:31.669 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:50:31.671+0000 7f06d4e00700 1 -- 172.21.15.27:0/4090325259 wait complete. 2024-04-16T05:50:31.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:31 smithi122 ceph-mon[80930]: from='client.34412 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:50:31.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:31 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1510241358' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:50:31.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:31 smithi027 ceph-mon[93635]: from='client.34412 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:50:31.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:31 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1510241358' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:50:32.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:32 smithi122 ceph-mon[80930]: pgmap v190: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:32.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:32 smithi122 ceph-mon[80930]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:50:32.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:32 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/4090325259' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:50:32.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:32 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:32.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:32 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:32.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:32 smithi027 ceph-mon[93635]: pgmap v190: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:32.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:32 smithi027 ceph-mon[93635]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:50:32.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:32 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/4090325259' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:50:32.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:32 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:32.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:32 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:34.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:34 smithi122 ceph-mon[80930]: pgmap v191: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:34.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:34 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:34.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:34 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:34.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:34 smithi027 ceph-mon[93635]: pgmap v191: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:34.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:34 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:34.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:34 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:36.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:36 smithi122 ceph-mon[80930]: pgmap v192: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:36.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:36 smithi027 ceph-mon[93635]: pgmap v192: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:38.465 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 354, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120854 2024-04-16T05:50:38.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 354, success, outsize: 120 2024-04-16T05:50:38.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 356, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 120936 2024-04-16T05:50:38.491 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 356, success, outsize: 120 2024-04-16T05:50:38.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 358, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121021 2024-04-16T05:50:38.517 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 358, success, outsize: 120 2024-04-16T05:50:38.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 360, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121112 2024-04-16T05:50:38.545 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 360, success, outsize: 120 2024-04-16T05:50:38.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 362, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121200 2024-04-16T05:50:38.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 362, success, outsize: 120 2024-04-16T05:50:38.603 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 364, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121282 2024-04-16T05:50:38.603 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 364, success, outsize: 120 2024-04-16T05:50:38.627 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 366, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121367 2024-04-16T05:50:38.628 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 366, success, outsize: 120 2024-04-16T05:50:38.653 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 368, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121458 2024-04-16T05:50:38.653 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 368, success, outsize: 120 2024-04-16T05:50:38.678 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 370, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121546 2024-04-16T05:50:38.678 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 370, success, outsize: 120 2024-04-16T05:50:38.701 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 372, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121628 2024-04-16T05:50:38.701 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 372, success, outsize: 120 2024-04-16T05:50:38.725 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 374, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121713 2024-04-16T05:50:38.725 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 374, success, outsize: 120 2024-04-16T05:50:38.750 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 376, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121804 2024-04-16T05:50:38.751 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 376, success, outsize: 120 2024-04-16T05:50:38.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:38 smithi122 ceph-mon[80930]: pgmap v193: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:38.777 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 378, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121892 2024-04-16T05:50:38.777 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 378, success, outsize: 120 2024-04-16T05:50:38.800 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 380, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 121974 2024-04-16T05:50:38.800 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 380, success, outsize: 120 2024-04-16T05:50:38.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:38 smithi027 ceph-mon[93635]: pgmap v193: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:38.824 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 382, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 122059 2024-04-16T05:50:38.824 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 382, success, outsize: 120 2024-04-16T05:50:38.849 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 384, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 122150 2024-04-16T05:50:38.849 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 384, success, outsize: 120 2024-04-16T05:50:38.874 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 386, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 122238 2024-04-16T05:50:38.874 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 386, success, outsize: 120 2024-04-16T05:50:38.897 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 388, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 122320 2024-04-16T05:50:38.898 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 388, success, outsize: 120 2024-04-16T05:50:38.921 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 390, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 122405 2024-04-16T05:50:38.921 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 390, success, outsize: 120 2024-04-16T05:50:38.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 392, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 122496 2024-04-16T05:50:38.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 392, success, outsize: 120 2024-04-16T05:50:38.971 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 394, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 122584 2024-04-16T05:50:38.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 394, success, outsize: 120 2024-04-16T05:50:38.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 396, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 122666 2024-04-16T05:50:38.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 396, success, outsize: 120 2024-04-16T05:50:39.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 398, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 122751 2024-04-16T05:50:39.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 398, success, outsize: 120 2024-04-16T05:50:39.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 400, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 122842 2024-04-16T05:50:39.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 400, success, outsize: 120 2024-04-16T05:50:39.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 402, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 122930 2024-04-16T05:50:39.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 402, success, outsize: 120 2024-04-16T05:50:39.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 404, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 123012 2024-04-16T05:50:39.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 404, success, outsize: 120 2024-04-16T05:50:39.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 406, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 123097 2024-04-16T05:50:39.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 406, success, outsize: 120 2024-04-16T05:50:39.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 408, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 123188 2024-04-16T05:50:39.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 408, success, outsize: 120 2024-04-16T05:50:40.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:40 smithi122 ceph-mon[80930]: pgmap v194: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:40.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:40 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:40.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:40 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:40.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:40 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:50:40.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:40 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:50:40.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:40 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:40.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:40 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:50:40.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:40 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:50:40.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:40 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:50:40.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:40 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:50:40.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:40 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:50:40.773 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:40 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi027.ejoytk"]}]: dispatch 2024-04-16T05:50:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:40 smithi027 ceph-mon[93635]: pgmap v194: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:50:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:50:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:50:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:50:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:50:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:50:40.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:50:40.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:40 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi027.ejoytk"]}]: dispatch 2024-04-16T05:50:41.701 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:41 smithi027 ceph-mon[93635]: Detected new or changed devices on smithi027 2024-04-16T05:50:41.701 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:41 smithi027 ceph-mon[93635]: Upgrade: It appears safe to stop mds.cephfs.smithi027.ejoytk 2024-04-16T05:50:42.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:41 smithi122 ceph-mon[80930]: Detected new or changed devices on smithi027 2024-04-16T05:50:42.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:41 smithi122 ceph-mon[80930]: Upgrade: It appears safe to stop mds.cephfs.smithi027.ejoytk 2024-04-16T05:50:42.670 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:42 smithi027 ceph-mon[93635]: pgmap v195: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:43.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:42 smithi122 ceph-mon[80930]: pgmap v195: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:44.003 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:43 smithi027 ceph-mon[93635]: Upgrade: Updating mds.cephfs.smithi027.ejoytk 2024-04-16T05:50:44.003 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:43 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:44.004 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:43 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.ejoytk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:50:44.004 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:43 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:50:44.004 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:43 smithi027 ceph-mon[93635]: Deploying daemon mds.cephfs.smithi027.ejoytk on smithi027 2024-04-16T05:50:44.004 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:43 smithi027 ceph-mon[93635]: pgmap v196: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:44.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:43 smithi122 ceph-mon[80930]: Upgrade: Updating mds.cephfs.smithi027.ejoytk 2024-04-16T05:50:44.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:43 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:44.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:43 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi027.ejoytk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:50:44.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:43 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:50:44.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:43 smithi122 ceph-mon[80930]: Deploying daemon mds.cephfs.smithi027.ejoytk on smithi027 2024-04-16T05:50:44.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:43 smithi122 ceph-mon[80930]: pgmap v196: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:44.928 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:50:45.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:44 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:50:45.898 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 410, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 123608 2024-04-16T05:50:45.898 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 410, success, outsize: 120 2024-04-16T05:50:45.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 522, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104494 2024-04-16T05:50:45.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 522, success, outsize: 120 2024-04-16T05:50:46.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:45 smithi027 ceph-mon[93635]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:46.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:45 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027[93631]: 2024-04-16T05:50:45.881+0000 7fdd83600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T05:50:46.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:45 smithi122 ceph-mon[80930]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:50:46.903 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 412, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 123609 2024-04-16T05:50:46.904 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 524, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104505 2024-04-16T05:50:46.904 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 412, success, outsize: 120 2024-04-16T05:50:46.905 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 524, success, outsize: 120 2024-04-16T05:50:47.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:46 smithi027 ceph-mon[93635]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T05:50:47.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:46 smithi027 ceph-mon[93635]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T05:50:47.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:46 smithi027 ceph-mon[93635]: osdmap e86: 6 total, 6 up, 6 in 2024-04-16T05:50:47.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:46 smithi027 ceph-mon[93635]: Standby daemon mds.cephfs.smithi122.lojjdp assigned to filesystem cephfs as rank 0 2024-04-16T05:50:47.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:46 smithi027 ceph-mon[93635]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T05:50:47.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:46 smithi027 ceph-mon[93635]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-16T05:50:47.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:46 smithi027 ceph-mon[93635]: fsmap cephfs:1/1 {0=cephfs.smithi122.lojjdp=up:replay} 2 up:standby 2024-04-16T05:50:47.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:46 smithi122 ceph-mon[80930]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T05:50:47.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:46 smithi122 ceph-mon[80930]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T05:50:47.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:46 smithi122 ceph-mon[80930]: osdmap e86: 6 total, 6 up, 6 in 2024-04-16T05:50:47.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:46 smithi122 ceph-mon[80930]: Standby daemon mds.cephfs.smithi122.lojjdp assigned to filesystem cephfs as rank 0 2024-04-16T05:50:47.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:46 smithi122 ceph-mon[80930]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T05:50:47.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:46 smithi122 ceph-mon[80930]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-16T05:50:47.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:46 smithi122 ceph-mon[80930]: fsmap cephfs:1/1 {0=cephfs.smithi122.lojjdp=up:replay} 2 up:standby 2024-04-16T05:50:48.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:47 smithi122 ceph-mon[80930]: reconnect by client.24301 192.168.0.1:0/790915989 after 0.00199995 2024-04-16T05:50:48.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:47 smithi122 ceph-mon[80930]: reconnect by client.14518 192.168.0.1:0/676551963 after 0.00299993 2024-04-16T05:50:48.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:47 smithi122 ceph-mon[80930]: mds.? [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] up:reconnect 2024-04-16T05:50:48.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:47 smithi122 ceph-mon[80930]: fsmap cephfs:1/1 {0=cephfs.smithi122.lojjdp=up:reconnect} 2 up:standby 2024-04-16T05:50:48.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:47 smithi122 ceph-mon[80930]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-16T05:50:48.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:47 smithi027 ceph-mon[93635]: reconnect by client.24301 192.168.0.1:0/790915989 after 0.00199995 2024-04-16T05:50:48.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:47 smithi027 ceph-mon[93635]: reconnect by client.14518 192.168.0.1:0/676551963 after 0.00299993 2024-04-16T05:50:48.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:47 smithi027 ceph-mon[93635]: mds.? [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] up:reconnect 2024-04-16T05:50:48.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:47 smithi027 ceph-mon[93635]: fsmap cephfs:1/1 {0=cephfs.smithi122.lojjdp=up:reconnect} 2 up:standby 2024-04-16T05:50:48.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:47 smithi027 ceph-mon[93635]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-16T05:50:49.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:48 smithi122 ceph-mon[80930]: mds.? [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] up:rejoin 2024-04-16T05:50:49.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:48 smithi122 ceph-mon[80930]: fsmap cephfs:1/1 {0=cephfs.smithi122.lojjdp=up:rejoin} 2 up:standby 2024-04-16T05:50:49.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:48 smithi122 ceph-mon[80930]: daemon mds.cephfs.smithi122.lojjdp is now active in filesystem cephfs as rank 0 2024-04-16T05:50:49.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:48 smithi027 ceph-mon[93635]: mds.? [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] up:rejoin 2024-04-16T05:50:49.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:48 smithi027 ceph-mon[93635]: fsmap cephfs:1/1 {0=cephfs.smithi122.lojjdp=up:rejoin} 2 up:standby 2024-04-16T05:50:49.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:48 smithi027 ceph-mon[93635]: daemon mds.cephfs.smithi122.lojjdp is now active in filesystem cephfs as rank 0 2024-04-16T05:50:50.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:49 smithi122 ceph-mon[80930]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T05:50:50.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:49 smithi122 ceph-mon[80930]: Cluster is now healthy 2024-04-16T05:50:50.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:49 smithi122 ceph-mon[80930]: mds.? [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] up:active 2024-04-16T05:50:50.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:49 smithi122 ceph-mon[80930]: fsmap cephfs:1 {0=cephfs.smithi122.lojjdp=up:active} 2 up:standby 2024-04-16T05:50:50.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:49 smithi122 ceph-mon[80930]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-16T05:50:50.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:49 smithi027 ceph-mon[93635]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T05:50:50.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:49 smithi027 ceph-mon[93635]: Cluster is now healthy 2024-04-16T05:50:50.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:49 smithi027 ceph-mon[93635]: mds.? [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] up:active 2024-04-16T05:50:50.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:49 smithi027 ceph-mon[93635]: fsmap cephfs:1 {0=cephfs.smithi122.lojjdp=up:active} 2 up:standby 2024-04-16T05:50:50.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:49 smithi027 ceph-mon[93635]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-16T05:50:52.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:52 smithi122 ceph-mon[80930]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 3.2 KiB/s rd, 2 op/s 2024-04-16T05:50:52.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:52 smithi027 ceph-mon[93635]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 3.2 KiB/s rd, 2 op/s 2024-04-16T05:50:54.362 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:54 smithi122 ceph-mon[80930]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 204 B/s wr, 4 op/s 2024-04-16T05:50:54.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:54 smithi027 ceph-mon[93635]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 204 B/s wr, 4 op/s 2024-04-16T05:50:56.937 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:56 smithi027 ceph-mon[93635]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 5.6 KiB/s rd, 204 B/s wr, 5 op/s 2024-04-16T05:50:56.937 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:56 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:56.937 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:56 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:56.938 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:56 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:50:56.938 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:56 smithi027 ceph-mon[93635]: mds.? [v2:172.21.15.27:6828/3219609959,v1:172.21.15.27:6829/3219609959] up:boot 2024-04-16T05:50:56.938 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:56 smithi027 ceph-mon[93635]: fsmap cephfs:1 {0=cephfs.smithi122.lojjdp=up:active} 3 up:standby 2024-04-16T05:50:56.938 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:56 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.ejoytk"}]: dispatch 2024-04-16T05:50:57.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:56 smithi122 ceph-mon[80930]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 5.6 KiB/s rd, 204 B/s wr, 5 op/s 2024-04-16T05:50:57.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:56 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:57.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:56 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:50:57.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:56 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:50:57.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:56 smithi122 ceph-mon[80930]: mds.? [v2:172.21.15.27:6828/3219609959,v1:172.21.15.27:6829/3219609959] up:boot 2024-04-16T05:50:57.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:56 smithi122 ceph-mon[80930]: fsmap cephfs:1 {0=cephfs.smithi122.lojjdp=up:active} 3 up:standby 2024-04-16T05:50:57.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:56 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi027.ejoytk"}]: dispatch 2024-04-16T05:50:58.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:58 smithi027 ceph-mon[93635]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 183 B/s wr, 4 op/s 2024-04-16T05:50:59.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:58 smithi122 ceph-mon[80930]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 183 B/s wr, 4 op/s 2024-04-16T05:51:00.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:50:59 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:51:00.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:50:59 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:51:01.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:00 smithi122 ceph-mon[80930]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 3.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-04-16T05:51:01.049 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:00 smithi027 ceph-mon[93635]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 3.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-04-16T05:51:01.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.841+0000 7fedd4000700 1 -- 172.21.15.27:0/2232169312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedcc0709a0 msgr2=0x7fedcc070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.841+0000 7fedd4000700 1 --2- 172.21.15.27:0/2232169312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedcc0709a0 0x7fedcc070e10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fedb8004940 tx=0x7fedb8009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.841+0000 7fedd4000700 1 -- 172.21.15.27:0/2232169312 shutdown_connections 2024-04-16T05:51:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.841+0000 7fedd4000700 1 --2- 172.21.15.27:0/2232169312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedcc0709a0 0x7fedcc070e10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.841+0000 7fedd4000700 1 --2- 172.21.15.27:0/2232169312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedcc071f50 0x7fedcc070460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.842+0000 7fedd4000700 1 -- 172.21.15.27:0/2232169312 >> 172.21.15.27:0/2232169312 conn(0x7fedcc0f8530 msgr2=0x7fedcc0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.842+0000 7fedd4000700 1 -- 172.21.15.27:0/2232169312 shutdown_connections 2024-04-16T05:51:01.841 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.842+0000 7fedd4000700 1 -- 172.21.15.27:0/2232169312 wait complete. 2024-04-16T05:51:01.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.843+0000 7fedd4000700 1 Processor -- start 2024-04-16T05:51:01.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.843+0000 7fedd4000700 1 -- start start 2024-04-16T05:51:01.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.843+0000 7fedd4000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedcc0709a0 0x7fedcc114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:01.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.843+0000 7fedd4000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedcc071f50 0x7fedcc114580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:01.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.843+0000 7fedd4000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedcc114b80 con 0x7fedcc0709a0 2024-04-16T05:51:01.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.843+0000 7fedd4000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedcc1153d0 con 0x7fedcc071f50 2024-04-16T05:51:01.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.844+0000 7fedd1600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedcc0709a0 0x7fedcc114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:01.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.844+0000 7fedd1600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedcc0709a0 0x7fedcc114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42374/0 (socket says 172.21.15.27:42374) 2024-04-16T05:51:01.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.844+0000 7fedd1600700 1 -- 172.21.15.27:0/4124907388 learned_addr learned my addr 172.21.15.27:0/4124907388 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:51:01.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.844+0000 7fedd0c00700 1 --2- 172.21.15.27:0/4124907388 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedcc071f50 0x7fedcc114580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:01.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.844+0000 7fedd1600700 1 -- 172.21.15.27:0/4124907388 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedcc071f50 msgr2=0x7fedcc114580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:01.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.844+0000 7fedd1600700 1 --2- 172.21.15.27:0/4124907388 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedcc071f50 0x7fedcc114580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:01.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.844+0000 7fedd1600700 1 -- 172.21.15.27:0/4124907388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb8005040 con 0x7fedcc0709a0 2024-04-16T05:51:01.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.845+0000 7fedd1600700 1 --2- 172.21.15.27:0/4124907388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedcc0709a0 0x7fedcc114040 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fedc0002a80 tx=0x7fedc0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:01.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.845+0000 7fedc9600700 1 -- 172.21.15.27:0/4124907388 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedc0007cc0 con 0x7fedcc0709a0 2024-04-16T05:51:01.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.845+0000 7fedd4000700 1 -- 172.21.15.27:0/4124907388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedcc1156b0 con 0x7fedcc0709a0 2024-04-16T05:51:01.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.845+0000 7fedc9600700 1 -- 172.21.15.27:0/4124907388 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fedc000f040 con 0x7fedcc0709a0 2024-04-16T05:51:01.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.845+0000 7fedc9600700 1 -- 172.21.15.27:0/4124907388 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedc000dad0 con 0x7fedcc0709a0 2024-04-16T05:51:01.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.845+0000 7fedd4000700 1 -- 172.21.15.27:0/4124907388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedcc115bf0 con 0x7fedcc0709a0 2024-04-16T05:51:01.845 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.847+0000 7fedd4000700 1 -- 172.21.15.27:0/4124907388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedcc05f5f0 con 0x7fedcc0709a0 2024-04-16T05:51:01.847 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.848+0000 7fedc9600700 1 -- 172.21.15.27:0/4124907388 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fedc000dc30 con 0x7fedcc0709a0 2024-04-16T05:51:01.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.849+0000 7fedc9600700 1 --2- 172.21.15.27:0/4124907388 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fedb406ec10 0x7fedb40710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:01.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.849+0000 7fedc9600700 1 -- 172.21.15.27:0/4124907388 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6822+0+0 (secure 0 0 0) 0x7fedc008eeb0 con 0x7fedcc0709a0 2024-04-16T05:51:01.848 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.850+0000 7fedd0c00700 1 --2- 172.21.15.27:0/4124907388 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fedb406ec10 0x7fedb40710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:01.849 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.850+0000 7fedd0c00700 1 --2- 172.21.15.27:0/4124907388 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fedb406ec10 0x7fedb40710c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fedb800b3b0 tx=0x7fedb8009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:01.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:01.853+0000 7fedc9600700 1 -- 172.21.15.27:0/4124907388 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fedc005b620 con 0x7fedcc0709a0 2024-04-16T05:51:02.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:01 smithi122 ceph-mon[80930]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 3.2 KiB/s rd, 170 B/s wr, 3 op/s 2024-04-16T05:51:02.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:01 smithi027 ceph-mon[93635]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 3.2 KiB/s rd, 170 B/s wr, 3 op/s 2024-04-16T05:51:02.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.128+0000 7fedd4000700 1 -- 172.21.15.27:0/4124907388 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fedcc115fd0 con 0x7fedb406ec10 2024-04-16T05:51:02.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.131+0000 7fedc9600700 1 -- 172.21.15.27:0/4124907388 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7fedcc115fd0 con 0x7fedb406ec10 2024-04-16T05:51:02.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.134+0000 7fedd4000700 1 -- 172.21.15.27:0/4124907388 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fedb406ec10 msgr2=0x7fedb40710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:02.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.134+0000 7fedd4000700 1 --2- 172.21.15.27:0/4124907388 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fedb406ec10 0x7fedb40710c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fedb800b3b0 tx=0x7fedb8009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:02.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.134+0000 7fedd4000700 1 -- 172.21.15.27:0/4124907388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedcc0709a0 msgr2=0x7fedcc114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:02.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.134+0000 7fedd4000700 1 --2- 172.21.15.27:0/4124907388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedcc0709a0 0x7fedcc114040 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fedc0002a80 tx=0x7fedc0004600 comp rx=0 tx=0).stop 2024-04-16T05:51:02.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.134+0000 7fedd4000700 1 -- 172.21.15.27:0/4124907388 shutdown_connections 2024-04-16T05:51:02.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.134+0000 7fedd4000700 1 --2- 172.21.15.27:0/4124907388 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fedb406ec10 0x7fedb40710c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:02.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.134+0000 7fedd4000700 1 --2- 172.21.15.27:0/4124907388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fedcc0709a0 0x7fedcc114040 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:02.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.134+0000 7fedd4000700 1 --2- 172.21.15.27:0/4124907388 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fedcc071f50 0x7fedcc114580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:02.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.134+0000 7fedd4000700 1 -- 172.21.15.27:0/4124907388 >> 172.21.15.27:0/4124907388 conn(0x7fedcc0f8530 msgr2=0x7fedcc0f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:02.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.135+0000 7fedd4000700 1 -- 172.21.15.27:0/4124907388 shutdown_connections 2024-04-16T05:51:02.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.135+0000 7fedd4000700 1 -- 172.21.15.27:0/4124907388 wait complete. 2024-04-16T05:51:02.150 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:51:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.303+0000 7f3df8000700 1 -- 172.21.15.27:0/3958641688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3df00fcdc0 msgr2=0x7f3df00fd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:02.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.303+0000 7f3df8000700 1 --2- 172.21.15.27:0/3958641688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3df00fcdc0 0x7f3df00fd1d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3de4004940 tx=0x7f3de4009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:02.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.303+0000 7f3df8000700 1 -- 172.21.15.27:0/3958641688 shutdown_connections 2024-04-16T05:51:02.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.303+0000 7f3df8000700 1 --2- 172.21.15.27:0/3958641688 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3df00fd710 0x7f3df00fe070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:02.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.303+0000 7f3df8000700 1 --2- 172.21.15.27:0/3958641688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3df00fcdc0 0x7f3df00fd1d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:02.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.303+0000 7f3df8000700 1 -- 172.21.15.27:0/3958641688 >> 172.21.15.27:0/3958641688 conn(0x7f3df00f8540 msgr2=0x7f3df00fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:02.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.303+0000 7f3df8000700 1 -- 172.21.15.27:0/3958641688 shutdown_connections 2024-04-16T05:51:02.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.303+0000 7f3df8000700 1 -- 172.21.15.27:0/3958641688 wait complete. 2024-04-16T05:51:02.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.304+0000 7f3df8000700 1 Processor -- start 2024-04-16T05:51:02.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.304+0000 7f3df8000700 1 -- start start 2024-04-16T05:51:02.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.305+0000 7f3df8000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3df00fcdc0 0x7f3df010fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:02.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.305+0000 7f3df8000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3df00fd710 0x7f3df0110280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:02.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.305+0000 7f3df8000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df0110880 con 0x7f3df00fcdc0 2024-04-16T05:51:02.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.305+0000 7f3df8000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df0110fc0 con 0x7f3df00fd710 2024-04-16T05:51:02.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.305+0000 7f3df5600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3df00fcdc0 0x7f3df010fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:02.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.305+0000 7f3df5600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3df00fcdc0 0x7f3df010fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42388/0 (socket says 172.21.15.27:42388) 2024-04-16T05:51:02.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.305+0000 7f3df5600700 1 -- 172.21.15.27:0/1775309585 learned_addr learned my addr 172.21.15.27:0/1775309585 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:51:02.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.306+0000 7f3df4c00700 1 --2- 172.21.15.27:0/1775309585 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3df00fd710 0x7f3df0110280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:02.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.306+0000 7f3df5600700 1 -- 172.21.15.27:0/1775309585 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3df00fd710 msgr2=0x7f3df0110280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:02.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.306+0000 7f3df5600700 1 --2- 172.21.15.27:0/1775309585 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3df00fd710 0x7f3df0110280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:02.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.306+0000 7f3df5600700 1 -- 172.21.15.27:0/1775309585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de4005040 con 0x7f3df00fcdc0 2024-04-16T05:51:02.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.306+0000 7f3df5600700 1 --2- 172.21.15.27:0/1775309585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3df00fcdc0 0x7f3df010fd40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3de400b970 tx=0x7f3de4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:02.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.307+0000 7f3ded600700 1 -- 172.21.15.27:0/1775309585 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3de400e040 con 0x7f3df00fcdc0 2024-04-16T05:51:02.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.307+0000 7f3ded600700 1 -- 172.21.15.27:0/1775309585 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3de400bbf0 con 0x7f3df00fcdc0 2024-04-16T05:51:02.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.307+0000 7f3ded600700 1 -- 172.21.15.27:0/1775309585 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3de401c630 con 0x7f3df00fcdc0 2024-04-16T05:51:02.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.307+0000 7f3df8000700 1 -- 172.21.15.27:0/1775309585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3df0111240 con 0x7f3df00fcdc0 2024-04-16T05:51:02.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.307+0000 7f3df8000700 1 -- 172.21.15.27:0/1775309585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3df0111800 con 0x7f3df00fcdc0 2024-04-16T05:51:02.307 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.308+0000 7f3df8000700 1 -- 172.21.15.27:0/1775309585 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3df0101a60 con 0x7f3df00fcdc0 2024-04-16T05:51:02.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.309+0000 7f3ded600700 1 -- 172.21.15.27:0/1775309585 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3de400bd60 con 0x7f3df00fcdc0 2024-04-16T05:51:02.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.310+0000 7f3ded600700 1 --2- 172.21.15.27:0/1775309585 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3dd806ec10 0x7f3dd80710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:02.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.310+0000 7f3ded600700 1 -- 172.21.15.27:0/1775309585 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6822+0+0 (secure 0 0 0) 0x7f3de408fe20 con 0x7f3df00fcdc0 2024-04-16T05:51:02.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.311+0000 7f3df4c00700 1 --2- 172.21.15.27:0/1775309585 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3dd806ec10 0x7f3dd80710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:02.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.311+0000 7f3df4c00700 1 --2- 172.21.15.27:0/1775309585 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3dd806ec10 0x7f3dd80710c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3df00fcb90 tx=0x7f3ddc009ee0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:02.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.316+0000 7f3ded600700 1 -- 172.21.15.27:0/1775309585 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3de405c590 con 0x7f3df00fcdc0 2024-04-16T05:51:02.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.569+0000 7f3df8000700 1 -- 172.21.15.27:0/1775309585 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3df0111ae0 con 0x7f3dd806ec10 2024-04-16T05:51:02.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.571+0000 7f3ded600700 1 -- 172.21.15.27:0/1775309585 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f3df0111ae0 con 0x7f3dd806ec10 2024-04-16T05:51:02.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.573+0000 7f3df8000700 1 -- 172.21.15.27:0/1775309585 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3dd806ec10 msgr2=0x7f3dd80710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:02.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.574+0000 7f3df8000700 1 --2- 172.21.15.27:0/1775309585 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3dd806ec10 0x7f3dd80710c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3df00fcb90 tx=0x7f3ddc009ee0 comp rx=0 tx=0).stop 2024-04-16T05:51:02.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.574+0000 7f3df8000700 1 -- 172.21.15.27:0/1775309585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3df00fcdc0 msgr2=0x7f3df010fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:02.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.574+0000 7f3df8000700 1 --2- 172.21.15.27:0/1775309585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3df00fcdc0 0x7f3df010fd40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3de400b970 tx=0x7f3de4009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:02.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.574+0000 7f3df8000700 1 -- 172.21.15.27:0/1775309585 shutdown_connections 2024-04-16T05:51:02.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.574+0000 7f3df8000700 1 --2- 172.21.15.27:0/1775309585 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3dd806ec10 0x7f3dd80710c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:02.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.574+0000 7f3df8000700 1 --2- 172.21.15.27:0/1775309585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3df00fcdc0 0x7f3df010fd40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:02.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.574+0000 7f3df8000700 1 --2- 172.21.15.27:0/1775309585 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3df00fd710 0x7f3df0110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:02.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.574+0000 7f3df8000700 1 -- 172.21.15.27:0/1775309585 >> 172.21.15.27:0/1775309585 conn(0x7f3df00f8540 msgr2=0x7f3df00f9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:02.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.574+0000 7f3df8000700 1 -- 172.21.15.27:0/1775309585 shutdown_connections 2024-04-16T05:51:02.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.574+0000 7f3df8000700 1 -- 172.21.15.27:0/1775309585 wait complete. 2024-04-16T05:51:02.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.733+0000 7fc4e0a00700 1 -- 172.21.15.27:0/1547443251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc100030 msgr2=0x7fc4dc102410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:02.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.733+0000 7fc4e0a00700 1 --2- 172.21.15.27:0/1547443251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc100030 0x7fc4dc102410 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8004940 tx=0x7fc4c8009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:02.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.733+0000 7fc4e0a00700 1 -- 172.21.15.27:0/1547443251 shutdown_connections 2024-04-16T05:51:02.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.733+0000 7fc4e0a00700 1 --2- 172.21.15.27:0/1547443251 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc100030 0x7fc4dc102410 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:02.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.733+0000 7fc4e0a00700 1 --2- 172.21.15.27:0/1547443251 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4dc066230 0x7fc4dc0ffaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:02.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.733+0000 7fc4e0a00700 1 -- 172.21.15.27:0/1547443251 >> 172.21.15.27:0/1547443251 conn(0x7fc4dc0f74c0 msgr2=0x7fc4dc0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:02.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.733+0000 7fc4e0a00700 1 -- 172.21.15.27:0/1547443251 shutdown_connections 2024-04-16T05:51:02.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.734+0000 7fc4e0a00700 1 -- 172.21.15.27:0/1547443251 wait complete. 2024-04-16T05:51:02.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.734+0000 7fc4e0a00700 1 Processor -- start 2024-04-16T05:51:02.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.735+0000 7fc4e0a00700 1 -- start start 2024-04-16T05:51:02.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.735+0000 7fc4e0a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc066230 0x7fc4dc0fd520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:02.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.735+0000 7fc4e0a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4dc100030 0x7fc4dc0fb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:02.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.735+0000 7fc4e0a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc0fbfa0 con 0x7fc4dc066230 2024-04-16T05:51:02.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.735+0000 7fc4e0a00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc0fc110 con 0x7fc4dc100030 2024-04-16T05:51:02.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.735+0000 7fc4d9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc066230 0x7fc4dc0fd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:02.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.735+0000 7fc4d9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc066230 0x7fc4dc0fd520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42406/0 (socket says 172.21.15.27:42406) 2024-04-16T05:51:02.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.735+0000 7fc4d9e00700 1 -- 172.21.15.27:0/920485898 learned_addr learned my addr 172.21.15.27:0/920485898 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:51:02.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.736+0000 7fc4d9400700 1 --2- 172.21.15.27:0/920485898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4dc100030 0x7fc4dc0fb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:02.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.736+0000 7fc4d9e00700 1 -- 172.21.15.27:0/920485898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4dc100030 msgr2=0x7fc4dc0fb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:02.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.736+0000 7fc4d9e00700 1 --2- 172.21.15.27:0/920485898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4dc100030 0x7fc4dc0fb9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:02.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.736+0000 7fc4d9e00700 1 -- 172.21.15.27:0/920485898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c8005040 con 0x7fc4dc066230 2024-04-16T05:51:02.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.736+0000 7fc4d9e00700 1 --2- 172.21.15.27:0/920485898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc066230 0x7fc4dc0fd520 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc001ce0 tx=0x7fc4cc009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:02.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.736+0000 7fc4d2000700 1 -- 172.21.15.27:0/920485898 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4cc00db60 con 0x7fc4dc066230 2024-04-16T05:51:02.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.736+0000 7fc4e0a00700 1 -- 172.21.15.27:0/920485898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4dc0fc3f0 con 0x7fc4dc066230 2024-04-16T05:51:02.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.736+0000 7fc4d2000700 1 -- 172.21.15.27:0/920485898 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc4cc00f040 con 0x7fc4dc066230 2024-04-16T05:51:02.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.736+0000 7fc4d2000700 1 -- 172.21.15.27:0/920485898 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4cc009790 con 0x7fc4dc066230 2024-04-16T05:51:02.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.737+0000 7fc4e0a00700 1 -- 172.21.15.27:0/920485898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4dc115a80 con 0x7fc4dc066230 2024-04-16T05:51:02.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.738+0000 7fc4e0a00700 1 -- 172.21.15.27:0/920485898 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4dc05f5f0 con 0x7fc4dc066230 2024-04-16T05:51:02.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.739+0000 7fc4d2000700 1 -- 172.21.15.27:0/920485898 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc4cc007920 con 0x7fc4dc066230 2024-04-16T05:51:02.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.740+0000 7fc4d2000700 1 --2- 172.21.15.27:0/920485898 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc4c006ec10 0x7fc4c00710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:02.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.740+0000 7fc4d2000700 1 -- 172.21.15.27:0/920485898 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6822+0+0 (secure 0 0 0) 0x7fc4cc08e590 con 0x7fc4dc066230 2024-04-16T05:51:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.741+0000 7fc4d9400700 1 --2- 172.21.15.27:0/920485898 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc4c006ec10 0x7fc4c00710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:02.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.742+0000 7fc4d9400700 1 --2- 172.21.15.27:0/920485898 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc4c006ec10 0x7fc4c00710c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc4c800b3b0 tx=0x7fc4c8009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:02.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.745+0000 7fc4d2000700 1 -- 172.21.15.27:0/920485898 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc4cc057200 con 0x7fc4dc066230 2024-04-16T05:51:02.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.986+0000 7fc4e0a00700 1 -- 172.21.15.27:0/920485898 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4dc0f8fb0 con 0x7fc4c006ec10 2024-04-16T05:51:02.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:02.997+0000 7fc4d2000700 1 -- 172.21.15.27:0/920485898 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7fc4dc0f8fb0 con 0x7fc4c006ec10 2024-04-16T05:51:02.997 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:51:02.997 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (7m) 0s ago 18m 13.5M - 0.25.0 c8568f914cd2 4ed63a8a6b33 2024-04-16T05:51:02.997 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (4m) 0s ago 18m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 4aa1f0006685 2024-04-16T05:51:02.997 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (4m) 58s ago 16m 6656k - 19.0.0-2884-g218a6914 a4e4811815e4 1c1912507d56 2024-04-16T05:51:02.997 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (7m) 0s ago 17m 74.0M - 9.4.7 954c08fa6188 f7ee2a1afd12 2024-04-16T05:51:02.997 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (8s) 0s ago 10m 11.0M - 19.0.0-2884-g218a6914 a4e4811815e4 b3c2178c537e 2024-04-16T05:51:02.997 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (39s) 0s ago 10m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 14b0c83f2891 2024-04-16T05:51:02.998 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (10m) 58s ago 10m 14.4M - 17.2.7-847-g45317f95 b333581ccdf5 f5e36815ddfa 2024-04-16T05:51:02.998 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (10m) 58s ago 10m 14.3M - 17.2.7-847-g45317f95 b333581ccdf5 755a73c65a7f 2024-04-16T05:51:02.998 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:8443,9283,8765 running (6m) 0s ago 19m 509M - 19.0.0-2884-g218a6914 a4e4811815e4 b69a0b08e12b 2024-04-16T05:51:02.998 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283,8765 running (6m) 58s ago 16m 453M - 19.0.0-2884-g218a6914 a4e4811815e4 32e27bcd7e42 2024-04-16T05:51:02.998 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (5m) 0s ago 20m 53.4M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 299ca337e4ca 2024-04-16T05:51:02.998 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (5m) 58s ago 16m 34.4M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5db5f3037ed0 2024-04-16T05:51:02.998 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (7m) 0s ago 18m 7994k - 1.5.0 0da6a335fe13 5e29e29868b0 2024-04-16T05:51:02.998 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (7m) 58s ago 16m 7910k - 1.5.0 0da6a335fe13 a360922bc621 2024-04-16T05:51:02.998 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (3m) 0s ago 15m 45.9M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 cef03b67bc5a 2024-04-16T05:51:02.998 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (2m) 0s ago 14m 44.8M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3a6ab1fc6e79 2024-04-16T05:51:02.998 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (2m) 0s ago 14m 43.3M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3524731d7277 2024-04-16T05:51:02.998 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (116s) 58s ago 13m 42.7M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 897d3579f929 2024-04-16T05:51:02.998 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (90s) 58s ago 13m 41.2M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 fa97758ff7c5 2024-04-16T05:51:02.999 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (62s) 58s ago 12m 11.2M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 fd12260a820d 2024-04-16T05:51:02.999 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (6m) 0s ago 17m 59.2M - 2.43.0 a07b618ecd1d e70f55ba5ec0 2024-04-16T05:51:02.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.000+0000 7fc4e0a00700 1 -- 172.21.15.27:0/920485898 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc4c006ec10 msgr2=0x7fc4c00710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:03.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.000+0000 7fc4e0a00700 1 --2- 172.21.15.27:0/920485898 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc4c006ec10 0x7fc4c00710c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc4c800b3b0 tx=0x7fc4c8009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:03.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.000+0000 7fc4e0a00700 1 -- 172.21.15.27:0/920485898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc066230 msgr2=0x7fc4dc0fd520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:03.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.000+0000 7fc4e0a00700 1 --2- 172.21.15.27:0/920485898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc066230 0x7fc4dc0fd520 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc001ce0 tx=0x7fc4cc009e60 comp rx=0 tx=0).stop 2024-04-16T05:51:03.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.000+0000 7fc4e0a00700 1 -- 172.21.15.27:0/920485898 shutdown_connections 2024-04-16T05:51:03.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.000+0000 7fc4e0a00700 1 --2- 172.21.15.27:0/920485898 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc4c006ec10 0x7fc4c00710c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:03.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.000+0000 7fc4e0a00700 1 --2- 172.21.15.27:0/920485898 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4dc066230 0x7fc4dc0fd520 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:03.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.000+0000 7fc4e0a00700 1 --2- 172.21.15.27:0/920485898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc4dc100030 0x7fc4dc0fb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:03.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.000+0000 7fc4e0a00700 1 -- 172.21.15.27:0/920485898 >> 172.21.15.27:0/920485898 conn(0x7fc4dc0f74c0 msgr2=0x7fc4dc0f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:03.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.000+0000 7fc4e0a00700 1 -- 172.21.15.27:0/920485898 shutdown_connections 2024-04-16T05:51:03.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.000+0000 7fc4e0a00700 1 -- 172.21.15.27:0/920485898 wait complete. 2024-04-16T05:51:03.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.162+0000 7f9b7de00700 1 -- 172.21.15.27:0/1573616245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b781002b0 msgr2=0x7f9b78100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:03.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.162+0000 7f9b7de00700 1 --2- 172.21.15.27:0/1573616245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b781002b0 0x7f9b78100720 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9b64004940 tx=0x7f9b64009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:03.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.162+0000 7f9b7de00700 1 -- 172.21.15.27:0/1573616245 shutdown_connections 2024-04-16T05:51:03.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.162+0000 7f9b7de00700 1 --2- 172.21.15.27:0/1573616245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b781002b0 0x7f9b78100720 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:03.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.162+0000 7f9b7de00700 1 --2- 172.21.15.27:0/1573616245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b780ff000 0x7f9b780ff410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:03.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.162+0000 7f9b7de00700 1 -- 172.21.15.27:0/1573616245 >> 172.21.15.27:0/1573616245 conn(0x7f9b780fa950 msgr2=0x7f9b780fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:03.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.162+0000 7f9b7de00700 1 -- 172.21.15.27:0/1573616245 shutdown_connections 2024-04-16T05:51:03.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.162+0000 7f9b7de00700 1 -- 172.21.15.27:0/1573616245 wait complete. 2024-04-16T05:51:03.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.163+0000 7f9b7de00700 1 Processor -- start 2024-04-16T05:51:03.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.163+0000 7f9b7de00700 1 -- start start 2024-04-16T05:51:03.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.163+0000 7f9b7de00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b780ff000 0x7f9b7810fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:03.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.163+0000 7f9b7de00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b781002b0 0x7f9b78110180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:03.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.163+0000 7f9b7de00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b78110780 con 0x7f9b780ff000 2024-04-16T05:51:03.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.163+0000 7f9b7de00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b78110fd0 con 0x7f9b781002b0 2024-04-16T05:51:03.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.164+0000 7f9b76a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b780ff000 0x7f9b7810fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:03.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.164+0000 7f9b76a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b780ff000 0x7f9b7810fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42412/0 (socket says 172.21.15.27:42412) 2024-04-16T05:51:03.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.164+0000 7f9b76a00700 1 -- 172.21.15.27:0/2798928617 learned_addr learned my addr 172.21.15.27:0/2798928617 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:51:03.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.164+0000 7f9b76000700 1 --2- 172.21.15.27:0/2798928617 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b781002b0 0x7f9b78110180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:03.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.164+0000 7f9b76a00700 1 -- 172.21.15.27:0/2798928617 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b781002b0 msgr2=0x7f9b78110180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:03.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.164+0000 7f9b76a00700 1 --2- 172.21.15.27:0/2798928617 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b781002b0 0x7f9b78110180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:03.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.164+0000 7f9b76a00700 1 -- 172.21.15.27:0/2798928617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b64005040 con 0x7f9b780ff000 2024-04-16T05:51:03.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.165+0000 7f9b76a00700 1 --2- 172.21.15.27:0/2798928617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b780ff000 0x7f9b7810fc40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9b6800c830 tx=0x7f9b68004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:03.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.165+0000 7f9b6ea00700 1 -- 172.21.15.27:0/2798928617 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b6800a760 con 0x7f9b780ff000 2024-04-16T05:51:03.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.165+0000 7f9b7de00700 1 -- 172.21.15.27:0/2798928617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b781112b0 con 0x7f9b780ff000 2024-04-16T05:51:03.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.165+0000 7f9b6ea00700 1 -- 172.21.15.27:0/2798928617 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b6800ad90 con 0x7f9b780ff000 2024-04-16T05:51:03.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.165+0000 7f9b6ea00700 1 -- 172.21.15.27:0/2798928617 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b680113a0 con 0x7f9b780ff000 2024-04-16T05:51:03.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.165+0000 7f9b7de00700 1 -- 172.21.15.27:0/2798928617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b781117f0 con 0x7f9b780ff000 2024-04-16T05:51:03.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.166+0000 7f9b7de00700 1 -- 172.21.15.27:0/2798928617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b7805f5f0 con 0x7f9b780ff000 2024-04-16T05:51:03.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.167+0000 7f9b6ea00700 1 -- 172.21.15.27:0/2798928617 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9b68012400 con 0x7f9b780ff000 2024-04-16T05:51:03.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.169+0000 7f9b6ea00700 1 --2- 172.21.15.27:0/2798928617 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9b5c06ec10 0x7f9b5c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:03.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.169+0000 7f9b6ea00700 1 -- 172.21.15.27:0/2798928617 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6822+0+0 (secure 0 0 0) 0x7f9b6808ea00 con 0x7f9b780ff000 2024-04-16T05:51:03.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.169+0000 7f9b76000700 1 --2- 172.21.15.27:0/2798928617 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9b5c06ec10 0x7f9b5c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:03.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.170+0000 7f9b76000700 1 --2- 172.21.15.27:0/2798928617 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9b5c06ec10 0x7f9b5c0710c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9b6400b970 tx=0x7f9b64009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:03.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.174+0000 7f9b6ea00700 1 -- 172.21.15.27:0/2798928617 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9b6805b1f0 con 0x7f9b780ff000 2024-04-16T05:51:03.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:03 smithi027 ceph-mon[93635]: from='client.34436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:51:03.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:03.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:03.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:03 smithi027 ceph-mon[93635]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:51:03.486 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.488+0000 7f9b7de00700 1 -- 172.21.15.27:0/2798928617 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f9b78111bc0 con 0x7f9b780ff000 2024-04-16T05:51:03.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.488+0000 7f9b6ea00700 1 -- 172.21.15.27:0/2798928617 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f9b6805ad80 con 0x7f9b780ff000 2024-04-16T05:51:03.487 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:51:03.487 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:51:03.487 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:51:03.487 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:51:03.487 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:51:03.487 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:51:03.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:51:03.488 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:51:03.488 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T05:51:03.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:51:03.488 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:51:03.488 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2, 2024-04-16T05:51:03.488 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:51:03.488 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:51:03.488 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:51:03.488 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 2, 2024-04-16T05:51:03.488 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 12 2024-04-16T05:51:03.488 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:51:03.488 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:51:03.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.491+0000 7f9b7de00700 1 -- 172.21.15.27:0/2798928617 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9b5c06ec10 msgr2=0x7f9b5c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:03.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.491+0000 7f9b7de00700 1 --2- 172.21.15.27:0/2798928617 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9b5c06ec10 0x7f9b5c0710c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9b6400b970 tx=0x7f9b64009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:03.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.491+0000 7f9b7de00700 1 -- 172.21.15.27:0/2798928617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b780ff000 msgr2=0x7f9b7810fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:03.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.491+0000 7f9b7de00700 1 --2- 172.21.15.27:0/2798928617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b780ff000 0x7f9b7810fc40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9b6800c830 tx=0x7f9b68004470 comp rx=0 tx=0).stop 2024-04-16T05:51:03.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.491+0000 7f9b7de00700 1 -- 172.21.15.27:0/2798928617 shutdown_connections 2024-04-16T05:51:03.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.491+0000 7f9b7de00700 1 --2- 172.21.15.27:0/2798928617 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9b5c06ec10 0x7f9b5c0710c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:03.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.491+0000 7f9b7de00700 1 --2- 172.21.15.27:0/2798928617 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9b780ff000 0x7f9b7810fc40 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:03.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.491+0000 7f9b7de00700 1 --2- 172.21.15.27:0/2798928617 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9b781002b0 0x7f9b78110180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:03.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.491+0000 7f9b7de00700 1 -- 172.21.15.27:0/2798928617 >> 172.21.15.27:0/2798928617 conn(0x7f9b780fa950 msgr2=0x7f9b780fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:03.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.491+0000 7f9b7de00700 1 -- 172.21.15.27:0/2798928617 shutdown_connections 2024-04-16T05:51:03.491 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.491+0000 7f9b7de00700 1 -- 172.21.15.27:0/2798928617 wait complete. 2024-04-16T05:51:03.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:03 smithi122 ceph-mon[80930]: from='client.34436 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:51:03.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:03.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:03.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:03 smithi122 ceph-mon[80930]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:51:03.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.647+0000 7f069b000700 1 -- 172.21.15.27:0/1036754838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0694071e50 msgr2=0x7f0694070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:03.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.647+0000 7f069b000700 1 --2- 172.21.15.27:0/1036754838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0694071e50 0x7f0694070350 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0684004940 tx=0x7f0684009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:03.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.647+0000 7f069b000700 1 -- 172.21.15.27:0/1036754838 shutdown_connections 2024-04-16T05:51:03.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.647+0000 7f069b000700 1 --2- 172.21.15.27:0/1036754838 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0694070890 0x7f0694070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:03.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.647+0000 7f069b000700 1 --2- 172.21.15.27:0/1036754838 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0694071e50 0x7f0694070350 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:03.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.647+0000 7f069b000700 1 -- 172.21.15.27:0/1036754838 >> 172.21.15.27:0/1036754838 conn(0x7f06940f8540 msgr2=0x7f06940fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:03.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.647+0000 7f069b000700 1 -- 172.21.15.27:0/1036754838 shutdown_connections 2024-04-16T05:51:03.647 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.648+0000 7f069b000700 1 -- 172.21.15.27:0/1036754838 wait complete. 2024-04-16T05:51:03.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.648+0000 7f069b000700 1 Processor -- start 2024-04-16T05:51:03.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.648+0000 7f069b000700 1 -- start start 2024-04-16T05:51:03.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.649+0000 7f069b000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0694070890 0x7f069410fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:03.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.649+0000 7f069b000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0694071e50 0x7f0694110260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:03.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.649+0000 7f069b000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0694110860 con 0x7f0694070890 2024-04-16T05:51:03.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.649+0000 7f069b000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0694110fa0 con 0x7f0694071e50 2024-04-16T05:51:03.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.649+0000 7f0693e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0694070890 0x7f069410fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:03.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.649+0000 7f0693e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0694070890 0x7f069410fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42434/0 (socket says 172.21.15.27:42434) 2024-04-16T05:51:03.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.649+0000 7f0693e00700 1 -- 172.21.15.27:0/1392952727 learned_addr learned my addr 172.21.15.27:0/1392952727 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:51:03.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.649+0000 7f0693400700 1 --2- 172.21.15.27:0/1392952727 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0694071e50 0x7f0694110260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:03.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.650+0000 7f0693e00700 1 -- 172.21.15.27:0/1392952727 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0694071e50 msgr2=0x7f0694110260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:03.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.650+0000 7f0693e00700 1 --2- 172.21.15.27:0/1392952727 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0694071e50 0x7f0694110260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:03.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.650+0000 7f0693e00700 1 -- 172.21.15.27:0/1392952727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0684005040 con 0x7f0694070890 2024-04-16T05:51:03.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.650+0000 7f0693e00700 1 --2- 172.21.15.27:0/1392952727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0694070890 0x7f069410fd20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f068400b970 tx=0x7f0684009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:03.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.650+0000 7f068be00700 1 -- 172.21.15.27:0/1392952727 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f068400e040 con 0x7f0694070890 2024-04-16T05:51:03.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.650+0000 7f068be00700 1 -- 172.21.15.27:0/1392952727 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0684004290 con 0x7f0694070890 2024-04-16T05:51:03.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.650+0000 7f068be00700 1 -- 172.21.15.27:0/1392952727 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f068401c630 con 0x7f0694070890 2024-04-16T05:51:03.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.650+0000 7f069b000700 1 -- 172.21.15.27:0/1392952727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0694111220 con 0x7f0694070890 2024-04-16T05:51:03.650 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.650+0000 7f069b000700 1 -- 172.21.15.27:0/1392952727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06941117e0 con 0x7f0694070890 2024-04-16T05:51:03.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.652+0000 7f069b000700 1 -- 172.21.15.27:0/1392952727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069405f5f0 con 0x7f0694070890 2024-04-16T05:51:03.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.653+0000 7f068be00700 1 -- 172.21.15.27:0/1392952727 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f068400bbf0 con 0x7f0694070890 2024-04-16T05:51:03.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.654+0000 7f068be00700 1 --2- 172.21.15.27:0/1392952727 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f067806ec10 0x7f06780710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:03.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.654+0000 7f068be00700 1 -- 172.21.15.27:0/1392952727 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6822+0+0 (secure 0 0 0) 0x7f068408fd10 con 0x7f0694070890 2024-04-16T05:51:03.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.654+0000 7f0693400700 1 --2- 172.21.15.27:0/1392952727 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f067806ec10 0x7f06780710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:03.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.655+0000 7f0693400700 1 --2- 172.21.15.27:0/1392952727 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f067806ec10 0x7f06780710c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0680004990 tx=0x7f0680009ee0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:03.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.659+0000 7f068be00700 1 -- 172.21.15.27:0/1392952727 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f068405c500 con 0x7f0694070890 2024-04-16T05:51:03.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.939+0000 7f069b000700 1 -- 172.21.15.27:0/1392952727 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0694107ae0 con 0x7f0694070890 2024-04-16T05:51:03.939 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.940+0000 7f068be00700 1 -- 172.21.15.27:0/1392952727 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+1893 (secure 0 0 0) 0x7f0684013e60 con 0x7f0694070890 2024-04-16T05:51:03.939 INFO:teuthology.orchestra.run.smithi027.stdout:e24 2024-04-16T05:51:03.939 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:51:03.940 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:51:03.940 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:51:03.940 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:51:03.940 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:51:03.940 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:51:03.940 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 23 2024-04-16T05:51:03.940 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:51:03.940 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:51:03.940 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:50:48.911395+0000 2024-04-16T05:51:03.940 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:51:03.940 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:51:03.940 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:51:03.940 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:51:03.940 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:51:03.941 INFO:teuthology.orchestra.run.smithi027.stdout:max_xattr_size 65536 2024-04-16T05:51:03.941 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:51:03.941 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:51:03.941 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 86 2024-04-16T05:51:03.941 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:51:03.941 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:51:03.941 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:51:03.941 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=24263} 2024-04-16T05:51:03.941 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:51:03.941 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:51:03.941 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:51:03.941 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:51:03.941 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:51:03.941 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:51:03.942 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:51:03.942 INFO:teuthology.orchestra.run.smithi027.stdout:bal_rank_mask -1 2024-04-16T05:51:03.942 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:51:03.942 INFO:teuthology.orchestra.run.smithi027.stdout:qdb_cluster leader: 24263 members: 24263 2024-04-16T05:51:03.942 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{0:24263} state up:active seq 160 join_fscid=1 addr [v2:172.21.15.122:6824/194115669,v1:172.21.15.122:6825/194115669] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:51:03.942 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:51:03.942 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:51:03.942 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:51:03.942 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:51:03.942 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:24273} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:51:03.942 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{-1:34400} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2831468256,v1:172.21.15.27:6827/2831468256] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:51:03.942 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{-1:34432} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6828/3219609959,v1:172.21.15.27:6829/3219609959] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:51:03.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.944+0000 7f069b000700 1 -- 172.21.15.27:0/1392952727 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f067806ec10 msgr2=0x7f06780710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:03.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.944+0000 7f069b000700 1 --2- 172.21.15.27:0/1392952727 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f067806ec10 0x7f06780710c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0680004990 tx=0x7f0680009ee0 comp rx=0 tx=0).stop 2024-04-16T05:51:03.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.944+0000 7f069b000700 1 -- 172.21.15.27:0/1392952727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0694070890 msgr2=0x7f069410fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:03.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.944+0000 7f069b000700 1 --2- 172.21.15.27:0/1392952727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0694070890 0x7f069410fd20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f068400b970 tx=0x7f0684009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:03.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.944+0000 7f069b000700 1 -- 172.21.15.27:0/1392952727 shutdown_connections 2024-04-16T05:51:03.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.944+0000 7f069b000700 1 --2- 172.21.15.27:0/1392952727 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f067806ec10 0x7f06780710c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:03.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.944+0000 7f069b000700 1 --2- 172.21.15.27:0/1392952727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0694070890 0x7f069410fd20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:03.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.944+0000 7f069b000700 1 --2- 172.21.15.27:0/1392952727 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0694071e50 0x7f0694110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:03.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.944+0000 7f069b000700 1 -- 172.21.15.27:0/1392952727 >> 172.21.15.27:0/1392952727 conn(0x7f06940f8540 msgr2=0x7f06940f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:03.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.945+0000 7f069b000700 1 -- 172.21.15.27:0/1392952727 shutdown_connections 2024-04-16T05:51:03.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:03.945+0000 7f069b000700 1 -- 172.21.15.27:0/1392952727 wait complete. 2024-04-16T05:51:03.945 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 24 2024-04-16T05:51:04.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.105+0000 7f9aab400700 1 -- 172.21.15.27:0/1653699546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa4070890 msgr2=0x7f9aa4070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:04.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.105+0000 7f9aab400700 1 --2- 172.21.15.27:0/1653699546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa4070890 0x7f9aa4070d00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9a90004940 tx=0x7f9a90009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:04.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.105+0000 7f9aab400700 1 -- 172.21.15.27:0/1653699546 shutdown_connections 2024-04-16T05:51:04.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.105+0000 7f9aab400700 1 --2- 172.21.15.27:0/1653699546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa4070890 0x7f9aa4070d00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:04.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.105+0000 7f9aab400700 1 --2- 172.21.15.27:0/1653699546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa4071e40 0x7f9aa4070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:04.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.105+0000 7f9aab400700 1 -- 172.21.15.27:0/1653699546 >> 172.21.15.27:0/1653699546 conn(0x7f9aa40f8590 msgr2=0x7f9aa40fa9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:04.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.105+0000 7f9aab400700 1 -- 172.21.15.27:0/1653699546 shutdown_connections 2024-04-16T05:51:04.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.105+0000 7f9aab400700 1 -- 172.21.15.27:0/1653699546 wait complete. 2024-04-16T05:51:04.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.106+0000 7f9aab400700 1 Processor -- start 2024-04-16T05:51:04.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.106+0000 7f9aab400700 1 -- start start 2024-04-16T05:51:04.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.107+0000 7f9aab400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa4070890 0x7f9aa410fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:04.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.107+0000 7f9aab400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa4071e40 0x7f9aa4110250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.107+0000 7f9aab400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa4110850 con 0x7f9aa4071e40 2024-04-16T05:51:04.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.107+0000 7f9aab400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa4110f90 con 0x7f9aa4070890 2024-04-16T05:51:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.107+0000 7f9aa3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa4071e40 0x7f9aa4110250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.107+0000 7f9aa3e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa4071e40 0x7f9aa4110250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33872/0 (socket says 172.21.15.27:33872) 2024-04-16T05:51:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.107+0000 7f9aa3e00700 1 -- 172.21.15.27:0/3678761788 learned_addr learned my addr 172.21.15.27:0/3678761788 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:51:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.107+0000 7f9aa8a00700 1 --2- 172.21.15.27:0/3678761788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa4070890 0x7f9aa410fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.108+0000 7f9aa3e00700 1 -- 172.21.15.27:0/3678761788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa4070890 msgr2=0x7f9aa410fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:04.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.108+0000 7f9aa3e00700 1 --2- 172.21.15.27:0/3678761788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa4070890 0x7f9aa410fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:04.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.108+0000 7f9aa3e00700 1 -- 172.21.15.27:0/3678761788 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a90005040 con 0x7f9aa4071e40 2024-04-16T05:51:04.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.108+0000 7f9aa3e00700 1 --2- 172.21.15.27:0/3678761788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa4071e40 0x7f9aa4110250 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9a90009150 tx=0x7f9a90009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:04.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.108+0000 7f9aa0c00700 1 -- 172.21.15.27:0/3678761788 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a9000e040 con 0x7f9aa4071e40 2024-04-16T05:51:04.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.108+0000 7f9aab400700 1 -- 172.21.15.27:0/3678761788 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa4111210 con 0x7f9aa4071e40 2024-04-16T05:51:04.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.108+0000 7f9aa0c00700 1 -- 172.21.15.27:0/3678761788 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a9000bcc0 con 0x7f9aa4071e40 2024-04-16T05:51:04.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.108+0000 7f9aa0c00700 1 -- 172.21.15.27:0/3678761788 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a9001c630 con 0x7f9aa4071e40 2024-04-16T05:51:04.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.109+0000 7f9aab400700 1 -- 172.21.15.27:0/3678761788 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa41117d0 con 0x7f9aa4071e40 2024-04-16T05:51:04.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.111+0000 7f9aab400700 1 -- 172.21.15.27:0/3678761788 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa405f5f0 con 0x7f9aa4071e40 2024-04-16T05:51:04.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.111+0000 7f9aa0c00700 1 -- 172.21.15.27:0/3678761788 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9a90013420 con 0x7f9aa4071e40 2024-04-16T05:51:04.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.113+0000 7f9aa0c00700 1 --2- 172.21.15.27:0/3678761788 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9a8c06ec00 0x7f9a8c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:04.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.113+0000 7f9aa0c00700 1 -- 172.21.15.27:0/3678761788 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6822+0+0 (secure 0 0 0) 0x7f9a9008f5a0 con 0x7f9aa4071e40 2024-04-16T05:51:04.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.114+0000 7f9aa8a00700 1 --2- 172.21.15.27:0/3678761788 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9a8c06ec00 0x7f9a8c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:04.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.114+0000 7f9aa8a00700 1 --2- 172.21.15.27:0/3678761788 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9a8c06ec00 0x7f9a8c0710b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9a98007fd0 tx=0x7f9a98009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:04.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.118+0000 7f9aa0c00700 1 -- 172.21.15.27:0/3678761788 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9a9005be50 con 0x7f9aa4071e40 2024-04-16T05:51:04.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:04 smithi027 ceph-mon[93635]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:51:04.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:04 smithi027 ceph-mon[93635]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 170 B/s wr, 2 op/s 2024-04-16T05:51:04.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:04 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/2798928617' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:51:04.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:04 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1392952727' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:51:04.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:04 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:04.381 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.384+0000 7f9aab400700 1 -- 172.21.15.27:0/3678761788 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9aa4111b20 con 0x7f9a8c06ec00 2024-04-16T05:51:04.383 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.386+0000 7f9aa0c00700 1 -- 172.21.15.27:0/3678761788 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f9aa4111b20 con 0x7f9a8c06ec00 2024-04-16T05:51:04.384 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:51:04.384 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T05:51:04.384 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:51:04.384 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:51:04.384 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [ 2024-04-16T05:51:04.384 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr", 2024-04-16T05:51:04.384 INFO:teuthology.orchestra.run.smithi027.stdout: "mon", 2024-04-16T05:51:04.384 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-16T05:51:04.384 INFO:teuthology.orchestra.run.smithi027.stdout: "osd" 2024-04-16T05:51:04.384 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-16T05:51:04.384 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "14/21 daemons upgraded", 2024-04-16T05:51:04.384 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T05:51:04.385 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:51:04.385 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:51:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.388+0000 7f9aab400700 1 -- 172.21.15.27:0/3678761788 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9a8c06ec00 msgr2=0x7f9a8c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.388+0000 7f9aab400700 1 --2- 172.21.15.27:0/3678761788 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9a8c06ec00 0x7f9a8c0710b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9a98007fd0 tx=0x7f9a98009040 comp rx=0 tx=0).stop 2024-04-16T05:51:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.388+0000 7f9aab400700 1 -- 172.21.15.27:0/3678761788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa4071e40 msgr2=0x7f9aa4110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.388+0000 7f9aab400700 1 --2- 172.21.15.27:0/3678761788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa4071e40 0x7f9aa4110250 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9a90009150 tx=0x7f9a90009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.388+0000 7f9aab400700 1 -- 172.21.15.27:0/3678761788 shutdown_connections 2024-04-16T05:51:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.388+0000 7f9aab400700 1 --2- 172.21.15.27:0/3678761788 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9a8c06ec00 0x7f9a8c0710b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:04.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.388+0000 7f9aab400700 1 --2- 172.21.15.27:0/3678761788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9aa4070890 0x7f9aa410fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:04.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.388+0000 7f9aab400700 1 --2- 172.21.15.27:0/3678761788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9aa4071e40 0x7f9aa4110250 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:04.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.388+0000 7f9aab400700 1 -- 172.21.15.27:0/3678761788 >> 172.21.15.27:0/3678761788 conn(0x7f9aa40f8590 msgr2=0x7f9aa40f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:04.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.389+0000 7f9aab400700 1 -- 172.21.15.27:0/3678761788 shutdown_connections 2024-04-16T05:51:04.387 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.389+0000 7f9aab400700 1 -- 172.21.15.27:0/3678761788 wait complete. 2024-04-16T05:51:04.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:04 smithi122 ceph-mon[80930]: from='client.34444 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:51:04.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:04 smithi122 ceph-mon[80930]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 170 B/s wr, 2 op/s 2024-04-16T05:51:04.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:04 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/2798928617' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:51:04.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:04 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1392952727' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:51:04.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:04 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:04.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.544+0000 7fe29cc00700 1 -- 172.21.15.27:0/2637048435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe298071e40 msgr2=0x7fe298070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:04.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.544+0000 7fe29cc00700 1 --2- 172.21.15.27:0/2637048435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe298071e40 0x7fe298070350 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe288004940 tx=0x7fe288009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.545+0000 7fe29cc00700 1 -- 172.21.15.27:0/2637048435 shutdown_connections 2024-04-16T05:51:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.545+0000 7fe29cc00700 1 --2- 172.21.15.27:0/2637048435 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe298070890 0x7fe298070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.545+0000 7fe29cc00700 1 --2- 172.21.15.27:0/2637048435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe298071e40 0x7fe298070350 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.545+0000 7fe29cc00700 1 -- 172.21.15.27:0/2637048435 >> 172.21.15.27:0/2637048435 conn(0x7fe2980f8530 msgr2=0x7fe2980fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.545+0000 7fe29cc00700 1 -- 172.21.15.27:0/2637048435 shutdown_connections 2024-04-16T05:51:04.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.545+0000 7fe29cc00700 1 -- 172.21.15.27:0/2637048435 wait complete. 2024-04-16T05:51:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.546+0000 7fe29cc00700 1 Processor -- start 2024-04-16T05:51:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.546+0000 7fe29cc00700 1 -- start start 2024-04-16T05:51:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.546+0000 7fe29cc00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe298070890 0x7fe29810fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.546+0000 7fe29cc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe298071e40 0x7fe2981101a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.546+0000 7fe29cc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2981107a0 con 0x7fe298071e40 2024-04-16T05:51:04.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.546+0000 7fe29cc00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe298110f80 con 0x7fe298070890 2024-04-16T05:51:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.547+0000 7fe295400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe298071e40 0x7fe2981101a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.547+0000 7fe295400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe298071e40 0x7fe2981101a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33892/0 (socket says 172.21.15.27:33892) 2024-04-16T05:51:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.547+0000 7fe295400700 1 -- 172.21.15.27:0/2898376761 learned_addr learned my addr 172.21.15.27:0/2898376761 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:51:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.547+0000 7fe295e00700 1 --2- 172.21.15.27:0/2898376761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe298070890 0x7fe29810fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.547+0000 7fe295400700 1 -- 172.21.15.27:0/2898376761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe298070890 msgr2=0x7fe29810fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.547+0000 7fe295400700 1 --2- 172.21.15.27:0/2898376761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe298070890 0x7fe29810fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:04.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.547+0000 7fe295400700 1 -- 172.21.15.27:0/2898376761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe288005040 con 0x7fe298071e40 2024-04-16T05:51:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.548+0000 7fe295400700 1 --2- 172.21.15.27:0/2898376761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe298071e40 0x7fe2981101a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe28400c8c0 tx=0x7fe2840044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.548+0000 7fe28e000700 1 -- 172.21.15.27:0/2898376761 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe28400a980 con 0x7fe298071e40 2024-04-16T05:51:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.548+0000 7fe28e000700 1 -- 172.21.15.27:0/2898376761 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe28400ce60 con 0x7fe298071e40 2024-04-16T05:51:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.548+0000 7fe29cc00700 1 -- 172.21.15.27:0/2898376761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe298111260 con 0x7fe298071e40 2024-04-16T05:51:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.548+0000 7fe28e000700 1 -- 172.21.15.27:0/2898376761 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe2840115e0 con 0x7fe298071e40 2024-04-16T05:51:04.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.548+0000 7fe29cc00700 1 -- 172.21.15.27:0/2898376761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2981117a0 con 0x7fe298071e40 2024-04-16T05:51:04.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.550+0000 7fe29cc00700 1 -- 172.21.15.27:0/2898376761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe29805f5f0 con 0x7fe298071e40 2024-04-16T05:51:04.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.551+0000 7fe28e000700 1 -- 172.21.15.27:0/2898376761 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe284011740 con 0x7fe298071e40 2024-04-16T05:51:04.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.552+0000 7fe28e000700 1 --2- 172.21.15.27:0/2898376761 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe27c06ec10 0x7fe27c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:04.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.552+0000 7fe28e000700 1 -- 172.21.15.27:0/2898376761 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6822+0+0 (secure 0 0 0) 0x7fe28408e990 con 0x7fe298071e40 2024-04-16T05:51:04.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.552+0000 7fe295e00700 1 --2- 172.21.15.27:0/2898376761 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe27c06ec10 0x7fe27c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:04.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.553+0000 7fe295e00700 1 --2- 172.21.15.27:0/2898376761 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe27c06ec10 0x7fe27c0710c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe288009fd0 tx=0x7fe288009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:04.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.557+0000 7fe28e000700 1 -- 172.21.15.27:0/2898376761 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe28405b240 con 0x7fe298071e40 2024-04-16T05:51:04.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.899+0000 7fe29cc00700 1 -- 172.21.15.27:0/2898376761 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fe298111b60 con 0x7fe298071e40 2024-04-16T05:51:04.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.899+0000 7fe28e000700 1 -- 172.21.15.27:0/2898376761 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fe28405add0 con 0x7fe298071e40 2024-04-16T05:51:04.897 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:51:04.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.902+0000 7fe29cc00700 1 -- 172.21.15.27:0/2898376761 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe27c06ec10 msgr2=0x7fe27c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:04.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.902+0000 7fe29cc00700 1 --2- 172.21.15.27:0/2898376761 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe27c06ec10 0x7fe27c0710c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe288009fd0 tx=0x7fe288009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:04.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.902+0000 7fe29cc00700 1 -- 172.21.15.27:0/2898376761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe298071e40 msgr2=0x7fe2981101a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:04.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.902+0000 7fe29cc00700 1 --2- 172.21.15.27:0/2898376761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe298071e40 0x7fe2981101a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe28400c8c0 tx=0x7fe2840044c0 comp rx=0 tx=0).stop 2024-04-16T05:51:04.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.902+0000 7fe29cc00700 1 -- 172.21.15.27:0/2898376761 shutdown_connections 2024-04-16T05:51:04.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.902+0000 7fe29cc00700 1 --2- 172.21.15.27:0/2898376761 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe27c06ec10 0x7fe27c0710c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:04.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.902+0000 7fe29cc00700 1 --2- 172.21.15.27:0/2898376761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe298070890 0x7fe29810fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:04.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.902+0000 7fe29cc00700 1 --2- 172.21.15.27:0/2898376761 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe298071e40 0x7fe2981101a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:04.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.902+0000 7fe29cc00700 1 -- 172.21.15.27:0/2898376761 >> 172.21.15.27:0/2898376761 conn(0x7fe2980f8530 msgr2=0x7fe2980f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:04.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.902+0000 7fe29cc00700 1 -- 172.21.15.27:0/2898376761 shutdown_connections 2024-04-16T05:51:04.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:04.902+0000 7fe29cc00700 1 -- 172.21.15.27:0/2898376761 wait complete. 2024-04-16T05:51:05.500 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:05 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:05.500 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:05 smithi027 ceph-mon[93635]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:51:05.500 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:05 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/2898376761' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:51:05.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:05 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:05.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:05 smithi122 ceph-mon[80930]: from='client.34456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:51:05.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:05 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/2898376761' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:51:06.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:06 smithi122 ceph-mon[80930]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 B/s wr, 0 op/s 2024-04-16T05:51:06.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:06 smithi027 ceph-mon[93635]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 B/s wr, 0 op/s 2024-04-16T05:51:08.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 414, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125011 2024-04-16T05:51:08.279 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 414, success, outsize: 120 2024-04-16T05:51:08.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 416, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125109 2024-04-16T05:51:08.308 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 416, success, outsize: 120 2024-04-16T05:51:08.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 418, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125210 2024-04-16T05:51:08.335 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 418, success, outsize: 120 2024-04-16T05:51:08.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 420, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125317 2024-04-16T05:51:08.367 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 420, success, outsize: 120 2024-04-16T05:51:08.400 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 422, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125421 2024-04-16T05:51:08.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 422, success, outsize: 120 2024-04-16T05:51:08.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 424, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125519 2024-04-16T05:51:08.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 424, success, outsize: 120 2024-04-16T05:51:08.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 426, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125620 2024-04-16T05:51:08.464 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 426, success, outsize: 120 2024-04-16T05:51:08.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 428, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125727 2024-04-16T05:51:08.501 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 428, success, outsize: 120 2024-04-16T05:51:08.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 430, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125831 2024-04-16T05:51:08.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 430, success, outsize: 120 2024-04-16T05:51:08.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 432, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 125929 2024-04-16T05:51:08.554 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 432, success, outsize: 120 2024-04-16T05:51:08.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:08 smithi027 ceph-mon[93635]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:51:08.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 434, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126030 2024-04-16T05:51:08.582 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 434, success, outsize: 120 2024-04-16T05:51:08.609 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 436, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126137 2024-04-16T05:51:08.610 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 436, success, outsize: 120 2024-04-16T05:51:08.638 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 438, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126241 2024-04-16T05:51:08.638 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 438, success, outsize: 120 2024-04-16T05:51:08.666 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 440, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126339 2024-04-16T05:51:08.667 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 440, success, outsize: 120 2024-04-16T05:51:08.697 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 442, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126440 2024-04-16T05:51:08.697 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 442, success, outsize: 120 2024-04-16T05:51:08.728 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 444, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126547 2024-04-16T05:51:08.728 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 444, success, outsize: 120 2024-04-16T05:51:08.757 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 446, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126651 2024-04-16T05:51:08.757 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 446, success, outsize: 120 2024-04-16T05:51:08.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:08 smithi122 ceph-mon[80930]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:51:08.781 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 448, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126749 2024-04-16T05:51:08.781 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 448, success, outsize: 120 2024-04-16T05:51:08.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 450, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126850 2024-04-16T05:51:08.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 450, success, outsize: 120 2024-04-16T05:51:08.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 452, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 126957 2024-04-16T05:51:08.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 452, success, outsize: 120 2024-04-16T05:51:08.877 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 454, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 127061 2024-04-16T05:51:08.877 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 454, success, outsize: 120 2024-04-16T05:51:08.908 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 456, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 127159 2024-04-16T05:51:08.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 456, success, outsize: 120 2024-04-16T05:51:08.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 458, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 127260 2024-04-16T05:51:08.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 458, success, outsize: 120 2024-04-16T05:51:08.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 460, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 127367 2024-04-16T05:51:08.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 460, success, outsize: 120 2024-04-16T05:51:09.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 462, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 127471 2024-04-16T05:51:09.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 462, success, outsize: 120 2024-04-16T05:51:09.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 464, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 127569 2024-04-16T05:51:09.044 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 464, success, outsize: 120 2024-04-16T05:51:09.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 466, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 127670 2024-04-16T05:51:09.077 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 466, success, outsize: 120 2024-04-16T05:51:09.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 468, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 127777 2024-04-16T05:51:09.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 468, success, outsize: 120 2024-04-16T05:51:10.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:10 smithi027 ceph-mon[93635]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:51:10.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:10 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:10.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:10 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:10.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:10 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:51:10.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:10 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:51:10.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:10 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:10.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:10 smithi122 ceph-mon[80930]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:51:10.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:10 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:10.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:10 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:10.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:10 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:51:10.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:10 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:51:10.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:10 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:11.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:11 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:51:11.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:11 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:51:11.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:11 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:51:11.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:11 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:51:11.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:11 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:51:11.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:11 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi122.lojjdp"]}]: dispatch 2024-04-16T05:51:11.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:11 smithi027 ceph-mon[93635]: Upgrade: It appears safe to stop mds.cephfs.smithi122.lojjdp 2024-04-16T05:51:11.576 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:11 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:51:11.576 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:11 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:51:11.576 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:11 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:51:11.576 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:11 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:51:11.576 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:11 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:51:11.576 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:11 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi122.lojjdp"]}]: dispatch 2024-04-16T05:51:11.576 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:11 smithi122 ceph-mon[80930]: Upgrade: It appears safe to stop mds.cephfs.smithi122.lojjdp 2024-04-16T05:51:12.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:12 smithi122 ceph-mon[80930]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:51:12.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:12 smithi027 ceph-mon[93635]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:51:13.995 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:13 smithi122 ceph-mon[80930]: Upgrade: Updating mds.cephfs.smithi122.lojjdp 2024-04-16T05:51:13.995 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:13.996 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.lojjdp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:51:13.996 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:13 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:51:13.996 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:13 smithi122 ceph-mon[80930]: Deploying daemon mds.cephfs.smithi122.lojjdp on smithi122 2024-04-16T05:51:13.996 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:13 smithi122 ceph-mon[80930]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:51:14.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:13 smithi027 ceph-mon[93635]: Upgrade: Updating mds.cephfs.smithi122.lojjdp 2024-04-16T05:51:14.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:14.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.lojjdp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:51:14.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:13 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:51:14.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:13 smithi027 ceph-mon[93635]: Deploying daemon mds.cephfs.smithi122.lojjdp on smithi122 2024-04-16T05:51:14.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:13 smithi027 ceph-mon[93635]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:51:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:51:15.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:51:15.667 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 526, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104874 2024-04-16T05:51:15.668 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 470, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 127860 2024-04-16T05:51:15.668 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 526, success, outsize: 120 2024-04-16T05:51:15.669 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 470, success, outsize: 120 2024-04-16T05:51:16.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:15 smithi122 ceph-mon[80930]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:51:16.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:15 smithi122 ceph-mon[80930]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T05:51:16.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:15 smithi122 ceph-mon[80930]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T05:51:16.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:15 smithi122 ceph-mon[80930]: osdmap e87: 6 total, 6 up, 6 in 2024-04-16T05:51:16.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:15 smithi122 ceph-mon[80930]: Standby daemon mds.cephfs.smithi122.rulewi assigned to filesystem cephfs as rank 0 2024-04-16T05:51:16.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:15 smithi122 ceph-mon[80930]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T05:51:16.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:15 smithi122 ceph-mon[80930]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-16T05:51:16.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:15 smithi122 ceph-mon[80930]: fsmap cephfs:1/1 {0=cephfs.smithi122.rulewi=up:replay} 2 up:standby 2024-04-16T05:51:16.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:15 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027[93631]: 2024-04-16T05:51:15.646+0000 7fdd83600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T05:51:16.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:15 smithi027 ceph-mon[93635]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 176 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:51:16.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:15 smithi027 ceph-mon[93635]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T05:51:16.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:15 smithi027 ceph-mon[93635]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T05:51:16.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:15 smithi027 ceph-mon[93635]: osdmap e87: 6 total, 6 up, 6 in 2024-04-16T05:51:16.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:15 smithi027 ceph-mon[93635]: Standby daemon mds.cephfs.smithi122.rulewi assigned to filesystem cephfs as rank 0 2024-04-16T05:51:16.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:15 smithi027 ceph-mon[93635]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T05:51:16.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:15 smithi027 ceph-mon[93635]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-16T05:51:16.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:15 smithi027 ceph-mon[93635]: fsmap cephfs:1/1 {0=cephfs.smithi122.rulewi=up:replay} 2 up:standby 2024-04-16T05:51:16.671 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 472, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 127861 2024-04-16T05:51:16.672 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 528, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 104882 2024-04-16T05:51:16.672 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 528, success, outsize: 120 2024-04-16T05:51:16.672 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 472, success, outsize: 120 2024-04-16T05:51:18.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:17 smithi122 ceph-mon[80930]: reconnect by client.24301 192.168.0.1:0/790915989 after 0.00399991 2024-04-16T05:51:18.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:17 smithi122 ceph-mon[80930]: reconnect by client.14518 192.168.0.1:0/676551963 after 0.00399991 2024-04-16T05:51:18.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:17 smithi122 ceph-mon[80930]: mds.? [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] up:reconnect 2024-04-16T05:51:18.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:17 smithi122 ceph-mon[80930]: fsmap cephfs:1/1 {0=cephfs.smithi122.rulewi=up:reconnect} 2 up:standby 2024-04-16T05:51:18.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:17 smithi122 ceph-mon[80930]: pgmap v215: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 op/s 2024-04-16T05:51:18.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:17 smithi027 ceph-mon[93635]: reconnect by client.24301 192.168.0.1:0/790915989 after 0.00399991 2024-04-16T05:51:18.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:17 smithi027 ceph-mon[93635]: reconnect by client.14518 192.168.0.1:0/676551963 after 0.00399991 2024-04-16T05:51:18.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:17 smithi027 ceph-mon[93635]: mds.? [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] up:reconnect 2024-04-16T05:51:18.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:17 smithi027 ceph-mon[93635]: fsmap cephfs:1/1 {0=cephfs.smithi122.rulewi=up:reconnect} 2 up:standby 2024-04-16T05:51:18.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:17 smithi027 ceph-mon[93635]: pgmap v215: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 op/s 2024-04-16T05:51:19.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:18 smithi122 ceph-mon[80930]: mds.? [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] up:rejoin 2024-04-16T05:51:19.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:18 smithi122 ceph-mon[80930]: fsmap cephfs:1/1 {0=cephfs.smithi122.rulewi=up:rejoin} 2 up:standby 2024-04-16T05:51:19.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:18 smithi122 ceph-mon[80930]: daemon mds.cephfs.smithi122.rulewi is now active in filesystem cephfs as rank 0 2024-04-16T05:51:19.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:18 smithi027 ceph-mon[93635]: mds.? [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] up:rejoin 2024-04-16T05:51:19.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:18 smithi027 ceph-mon[93635]: fsmap cephfs:1/1 {0=cephfs.smithi122.rulewi=up:rejoin} 2 up:standby 2024-04-16T05:51:19.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:18 smithi027 ceph-mon[93635]: daemon mds.cephfs.smithi122.rulewi is now active in filesystem cephfs as rank 0 2024-04-16T05:51:20.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:19 smithi122 ceph-mon[80930]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T05:51:20.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:19 smithi122 ceph-mon[80930]: Cluster is now healthy 2024-04-16T05:51:20.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:19 smithi122 ceph-mon[80930]: mds.? [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] up:active 2024-04-16T05:51:20.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:19 smithi122 ceph-mon[80930]: fsmap cephfs:1 {0=cephfs.smithi122.rulewi=up:active} 2 up:standby 2024-04-16T05:51:20.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:19 smithi122 ceph-mon[80930]: pgmap v216: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-04-16T05:51:20.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:19 smithi027 ceph-mon[93635]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T05:51:20.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:19 smithi027 ceph-mon[93635]: Cluster is now healthy 2024-04-16T05:51:20.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:19 smithi027 ceph-mon[93635]: mds.? [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] up:active 2024-04-16T05:51:20.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:19 smithi027 ceph-mon[93635]: fsmap cephfs:1 {0=cephfs.smithi122.rulewi=up:active} 2 up:standby 2024-04-16T05:51:20.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:19 smithi027 ceph-mon[93635]: pgmap v216: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-04-16T05:51:22.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:22 smithi122 ceph-mon[80930]: pgmap v217: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 2 op/s 2024-04-16T05:51:22.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:22 smithi027 ceph-mon[93635]: pgmap v217: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 3.3 KiB/s rd, 2 op/s 2024-04-16T05:51:24.447 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:24 smithi122 ceph-mon[80930]: pgmap v218: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 5.1 KiB/s rd, 204 B/s wr, 4 op/s 2024-04-16T05:51:24.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:24 smithi027 ceph-mon[93635]: pgmap v218: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 5.1 KiB/s rd, 204 B/s wr, 4 op/s 2024-04-16T05:51:26.149 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:26 smithi122 ceph-mon[80930]: pgmap v219: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 5.7 KiB/s rd, 204 B/s wr, 5 op/s 2024-04-16T05:51:26.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:26 smithi027 ceph-mon[93635]: pgmap v219: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 5.7 KiB/s rd, 204 B/s wr, 5 op/s 2024-04-16T05:51:27.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:27 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:27.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:27 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:27.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:27 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:51:27.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:27 smithi122 ceph-mon[80930]: mds.? [v2:172.21.15.122:6824/97103507,v1:172.21.15.122:6825/97103507] up:boot 2024-04-16T05:51:27.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:27 smithi122 ceph-mon[80930]: fsmap cephfs:1 {0=cephfs.smithi122.rulewi=up:active} 3 up:standby 2024-04-16T05:51:27.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:27 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.lojjdp"}]: dispatch 2024-04-16T05:51:27.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:27 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:27.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:27 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:27.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:27 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:51:27.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:27 smithi027 ceph-mon[93635]: mds.? [v2:172.21.15.122:6824/97103507,v1:172.21.15.122:6825/97103507] up:boot 2024-04-16T05:51:27.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:27 smithi027 ceph-mon[93635]: fsmap cephfs:1 {0=cephfs.smithi122.rulewi=up:active} 3 up:standby 2024-04-16T05:51:27.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:27 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.lojjdp"}]: dispatch 2024-04-16T05:51:28.770 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:28 smithi122 ceph-mon[80930]: pgmap v220: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 179 B/s wr, 4 op/s 2024-04-16T05:51:28.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:28 smithi027 ceph-mon[93635]: pgmap v220: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 5.0 KiB/s rd, 179 B/s wr, 4 op/s 2024-04-16T05:51:29.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:51:29.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:51:30.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:30 smithi122 ceph-mon[80930]: pgmap v221: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 3.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-04-16T05:51:30.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:30 smithi027 ceph-mon[93635]: pgmap v221: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 3.7 KiB/s rd, 170 B/s wr, 3 op/s 2024-04-16T05:51:32.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:31 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:32.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:31 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:32.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:31 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:32.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:31 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:33.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:32 smithi122 ceph-mon[80930]: pgmap v222: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 3.2 KiB/s rd, 170 B/s wr, 3 op/s 2024-04-16T05:51:33.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:32 smithi027 ceph-mon[93635]: pgmap v222: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 3.2 KiB/s rd, 170 B/s wr, 3 op/s 2024-04-16T05:51:34.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:33 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:34.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:33 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:34.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:33 smithi122 ceph-mon[80930]: pgmap v223: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 170 B/s wr, 1 op/s 2024-04-16T05:51:34.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:33 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:34.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:33 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:34.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:33 smithi027 ceph-mon[93635]: pgmap v223: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 170 B/s wr, 1 op/s 2024-04-16T05:51:35.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.078+0000 7f94f7400700 1 -- 172.21.15.27:0/2918843546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94f00fe050 msgr2=0x7f94f00fe4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:35.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.078+0000 7f94f7400700 1 --2- 172.21.15.27:0/2918843546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94f00fe050 0x7f94f00fe4c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f94dc004880 tx=0x7f94dc009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:35.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.078+0000 7f94f7400700 1 -- 172.21.15.27:0/2918843546 shutdown_connections 2024-04-16T05:51:35.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.078+0000 7f94f7400700 1 --2- 172.21.15.27:0/2918843546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94f00fe050 0x7f94f00fe4c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:35.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.078+0000 7f94f7400700 1 --2- 172.21.15.27:0/2918843546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94f00fce50 0x7f94f00fd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:35.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.078+0000 7f94f7400700 1 -- 172.21.15.27:0/2918843546 >> 172.21.15.27:0/2918843546 conn(0x7f94f00f8540 msgr2=0x7f94f00fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:35.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.078+0000 7f94f7400700 1 -- 172.21.15.27:0/2918843546 shutdown_connections 2024-04-16T05:51:35.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.078+0000 7f94f7400700 1 -- 172.21.15.27:0/2918843546 wait complete. 2024-04-16T05:51:35.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.079+0000 7f94f7400700 1 Processor -- start 2024-04-16T05:51:35.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.079+0000 7f94f7400700 1 -- start start 2024-04-16T05:51:35.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.079+0000 7f94f7400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94f00fce50 0x7f94f010fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:35.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.079+0000 7f94f7400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94f00fe050 0x7f94f0110100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:35.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.079+0000 7f94f7400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f0110700 con 0x7f94f00fce50 2024-04-16T05:51:35.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.079+0000 7f94f7400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f0110870 con 0x7f94f00fe050 2024-04-16T05:51:35.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.080+0000 7f94f4a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94f00fce50 0x7f94f010fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:35.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.080+0000 7f94f4a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94f00fce50 0x7f94f010fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48418/0 (socket says 172.21.15.27:48418) 2024-04-16T05:51:35.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.080+0000 7f94f4a00700 1 -- 172.21.15.27:0/1600750048 learned_addr learned my addr 172.21.15.27:0/1600750048 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:51:35.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.080+0000 7f94efe00700 1 --2- 172.21.15.27:0/1600750048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94f00fe050 0x7f94f0110100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:35.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.080+0000 7f94f4a00700 1 -- 172.21.15.27:0/1600750048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94f00fe050 msgr2=0x7f94f0110100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:35.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.080+0000 7f94f4a00700 1 --2- 172.21.15.27:0/1600750048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94f00fe050 0x7f94f0110100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:35.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.080+0000 7f94f4a00700 1 -- 172.21.15.27:0/1600750048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94dc005040 con 0x7f94f00fce50 2024-04-16T05:51:35.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.081+0000 7f94f4a00700 1 --2- 172.21.15.27:0/1600750048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94f00fce50 0x7f94f010fbc0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f94e400eb40 tx=0x7f94e400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:35.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.081+0000 7f94ecc00700 1 -- 172.21.15.27:0/1600750048 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94e4007cc0 con 0x7f94f00fce50 2024-04-16T05:51:35.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.081+0000 7f94f7400700 1 -- 172.21.15.27:0/1600750048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f01152d0 con 0x7f94f00fce50 2024-04-16T05:51:35.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.081+0000 7f94ecc00700 1 -- 172.21.15.27:0/1600750048 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f94e400f040 con 0x7f94f00fce50 2024-04-16T05:51:35.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.081+0000 7f94ecc00700 1 -- 172.21.15.27:0/1600750048 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94e400cb50 con 0x7f94f00fce50 2024-04-16T05:51:35.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.081+0000 7f94f7400700 1 -- 172.21.15.27:0/1600750048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f0115890 con 0x7f94f00fce50 2024-04-16T05:51:35.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.082+0000 7f94f7400700 1 -- 172.21.15.27:0/1600750048 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f01019e0 con 0x7f94f00fce50 2024-04-16T05:51:35.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.084+0000 7f94ecc00700 1 -- 172.21.15.27:0/1600750048 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f94e400ccb0 con 0x7f94f00fce50 2024-04-16T05:51:35.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.085+0000 7f94ecc00700 1 --2- 172.21.15.27:0/1600750048 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f94d806ec00 0x7f94d80710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:35.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.085+0000 7f94ecc00700 1 -- 172.21.15.27:0/1600750048 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6908+0+0 (secure 0 0 0) 0x7f94e408e3a0 con 0x7f94f00fce50 2024-04-16T05:51:35.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.085+0000 7f94efe00700 1 --2- 172.21.15.27:0/1600750048 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f94d806ec00 0x7f94d80710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:35.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.086+0000 7f94efe00700 1 --2- 172.21.15.27:0/1600750048 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f94d806ec00 0x7f94d80710b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f94dc009fd0 tx=0x7f94dc009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:35.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.090+0000 7f94ecc00700 1 -- 172.21.15.27:0/1600750048 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f94e405ac10 con 0x7f94f00fce50 2024-04-16T05:51:35.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.342+0000 7f94f7400700 1 -- 172.21.15.27:0/1600750048 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f94f002b410 con 0x7f94d806ec00 2024-04-16T05:51:35.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.346+0000 7f94ecc00700 1 -- 172.21.15.27:0/1600750048 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f94f002b410 con 0x7f94d806ec00 2024-04-16T05:51:35.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.349+0000 7f94f7400700 1 -- 172.21.15.27:0/1600750048 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f94d806ec00 msgr2=0x7f94d80710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:35.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.349+0000 7f94f7400700 1 --2- 172.21.15.27:0/1600750048 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f94d806ec00 0x7f94d80710b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f94dc009fd0 tx=0x7f94dc009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:35.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.349+0000 7f94f7400700 1 -- 172.21.15.27:0/1600750048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94f00fce50 msgr2=0x7f94f010fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:35.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.349+0000 7f94f7400700 1 --2- 172.21.15.27:0/1600750048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94f00fce50 0x7f94f010fbc0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f94e400eb40 tx=0x7f94e400b040 comp rx=0 tx=0).stop 2024-04-16T05:51:35.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.350+0000 7f94f7400700 1 -- 172.21.15.27:0/1600750048 shutdown_connections 2024-04-16T05:51:35.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.350+0000 7f94f7400700 1 --2- 172.21.15.27:0/1600750048 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f94d806ec00 0x7f94d80710b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:35.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.350+0000 7f94f7400700 1 --2- 172.21.15.27:0/1600750048 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f94f00fce50 0x7f94f010fbc0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:35.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.350+0000 7f94f7400700 1 --2- 172.21.15.27:0/1600750048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f94f00fe050 0x7f94f0110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:35.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.350+0000 7f94f7400700 1 -- 172.21.15.27:0/1600750048 >> 172.21.15.27:0/1600750048 conn(0x7f94f00f8540 msgr2=0x7f94f00f9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:35.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.350+0000 7f94f7400700 1 -- 172.21.15.27:0/1600750048 shutdown_connections 2024-04-16T05:51:35.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.350+0000 7f94f7400700 1 -- 172.21.15.27:0/1600750048 wait complete. 2024-04-16T05:51:35.365 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:51:35.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.512+0000 7f4c65600700 1 -- 172.21.15.27:0/2978436243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c601002d0 msgr2=0x7f4c60100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:35.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.512+0000 7f4c65600700 1 --2- 172.21.15.27:0/2978436243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c601002d0 0x7f4c60100740 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c004940 tx=0x7f4c4c009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:35.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.512+0000 7f4c65600700 1 -- 172.21.15.27:0/2978436243 shutdown_connections 2024-04-16T05:51:35.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.512+0000 7f4c65600700 1 --2- 172.21.15.27:0/2978436243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c601002d0 0x7f4c60100740 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:35.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.512+0000 7f4c65600700 1 --2- 172.21.15.27:0/2978436243 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c600ff020 0x7f4c600ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:35.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.512+0000 7f4c65600700 1 -- 172.21.15.27:0/2978436243 >> 172.21.15.27:0/2978436243 conn(0x7f4c600fa950 msgr2=0x7f4c600fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:35.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.513+0000 7f4c65600700 1 -- 172.21.15.27:0/2978436243 shutdown_connections 2024-04-16T05:51:35.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.513+0000 7f4c65600700 1 -- 172.21.15.27:0/2978436243 wait complete. 2024-04-16T05:51:35.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.514+0000 7f4c65600700 1 Processor -- start 2024-04-16T05:51:35.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.514+0000 7f4c65600700 1 -- start start 2024-04-16T05:51:35.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.514+0000 7f4c65600700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c600ff020 0x7f4c6010fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:35.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.514+0000 7f4c65600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c601002d0 0x7f4c601100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:35.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.514+0000 7f4c65600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c601106b0 con 0x7f4c601002d0 2024-04-16T05:51:35.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.514+0000 7f4c65600700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c60110820 con 0x7f4c600ff020 2024-04-16T05:51:35.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.514+0000 7f4c5e000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c601002d0 0x7f4c601100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:35.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.514+0000 7f4c5e000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c601002d0 0x7f4c601100b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48442/0 (socket says 172.21.15.27:48442) 2024-04-16T05:51:35.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.514+0000 7f4c5e000700 1 -- 172.21.15.27:0/1637592256 learned_addr learned my addr 172.21.15.27:0/1637592256 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:51:35.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.515+0000 7f4c5e000700 1 -- 172.21.15.27:0/1637592256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c600ff020 msgr2=0x7f4c6010fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:35.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.515+0000 7f4c5ea00700 1 --2- 172.21.15.27:0/1637592256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c600ff020 0x7f4c6010fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:35.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.515+0000 7f4c5e000700 1 --2- 172.21.15.27:0/1637592256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c600ff020 0x7f4c6010fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:35.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.515+0000 7f4c5e000700 1 -- 172.21.15.27:0/1637592256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4c005040 con 0x7f4c601002d0 2024-04-16T05:51:35.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.515+0000 7f4c5e000700 1 --2- 172.21.15.27:0/1637592256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c601002d0 0x7f4c601100b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00b3b0 tx=0x7f4c4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:35.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.515+0000 7f4c56a00700 1 -- 172.21.15.27:0/1637592256 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c4c00e040 con 0x7f4c601002d0 2024-04-16T05:51:35.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.516+0000 7f4c56a00700 1 -- 172.21.15.27:0/1637592256 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c4c004290 con 0x7f4c601002d0 2024-04-16T05:51:35.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.516+0000 7f4c56a00700 1 -- 172.21.15.27:0/1637592256 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c4c01c630 con 0x7f4c601002d0 2024-04-16T05:51:35.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.516+0000 7f4c65600700 1 -- 172.21.15.27:0/1637592256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c60115220 con 0x7f4c601002d0 2024-04-16T05:51:35.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.516+0000 7f4c65600700 1 -- 172.21.15.27:0/1637592256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c601157e0 con 0x7f4c601002d0 2024-04-16T05:51:35.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.517+0000 7f4c65600700 1 -- 172.21.15.27:0/1637592256 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c6004c710 con 0x7f4c601002d0 2024-04-16T05:51:35.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.518+0000 7f4c56a00700 1 -- 172.21.15.27:0/1637592256 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4c4c00bbf0 con 0x7f4c601002d0 2024-04-16T05:51:35.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.519+0000 7f4c56a00700 1 --2- 172.21.15.27:0/1637592256 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4c4406ec00 0x7f4c440710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:35.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.519+0000 7f4c56a00700 1 -- 172.21.15.27:0/1637592256 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6908+0+0 (secure 0 0 0) 0x7f4c4c08fe30 con 0x7f4c601002d0 2024-04-16T05:51:35.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.519+0000 7f4c5ea00700 1 --2- 172.21.15.27:0/1637592256 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4c4406ec00 0x7f4c440710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:35.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.520+0000 7f4c5ea00700 1 --2- 172.21.15.27:0/1637592256 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4c4406ec00 0x7f4c440710b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4c500041d0 tx=0x7f4c50009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:35.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.524+0000 7f4c56a00700 1 -- 172.21.15.27:0/1637592256 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4c4c05c5d0 con 0x7f4c601002d0 2024-04-16T05:51:35.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.777+0000 7f4c65600700 1 -- 172.21.15.27:0/1637592256 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c601045f0 con 0x7f4c4406ec00 2024-04-16T05:51:35.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.780+0000 7f4c56a00700 1 -- 172.21.15.27:0/1637592256 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f4c601045f0 con 0x7f4c4406ec00 2024-04-16T05:51:35.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.783+0000 7f4c65600700 1 -- 172.21.15.27:0/1637592256 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4c4406ec00 msgr2=0x7f4c440710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:35.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.783+0000 7f4c65600700 1 --2- 172.21.15.27:0/1637592256 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4c4406ec00 0x7f4c440710b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4c500041d0 tx=0x7f4c50009040 comp rx=0 tx=0).stop 2024-04-16T05:51:35.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.783+0000 7f4c65600700 1 -- 172.21.15.27:0/1637592256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c601002d0 msgr2=0x7f4c601100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:35.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.783+0000 7f4c65600700 1 --2- 172.21.15.27:0/1637592256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c601002d0 0x7f4c601100b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c00b3b0 tx=0x7f4c4c009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:35.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.783+0000 7f4c65600700 1 -- 172.21.15.27:0/1637592256 shutdown_connections 2024-04-16T05:51:35.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.783+0000 7f4c65600700 1 --2- 172.21.15.27:0/1637592256 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4c4406ec00 0x7f4c440710b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:35.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.783+0000 7f4c65600700 1 --2- 172.21.15.27:0/1637592256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c600ff020 0x7f4c6010fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:35.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.783+0000 7f4c65600700 1 --2- 172.21.15.27:0/1637592256 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c601002d0 0x7f4c601100b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:35.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.783+0000 7f4c65600700 1 -- 172.21.15.27:0/1637592256 >> 172.21.15.27:0/1637592256 conn(0x7f4c600fa950 msgr2=0x7f4c600fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:35.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.783+0000 7f4c65600700 1 -- 172.21.15.27:0/1637592256 shutdown_connections 2024-04-16T05:51:35.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.784+0000 7f4c65600700 1 -- 172.21.15.27:0/1637592256 wait complete. 2024-04-16T05:51:35.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.943+0000 7f76fdc00700 1 -- 172.21.15.27:0/1098976610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76f81002d0 msgr2=0x7f76f8100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:35.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.943+0000 7f76fdc00700 1 --2- 172.21.15.27:0/1098976610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76f81002d0 0x7f76f8100740 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f76e4004940 tx=0x7f76e4009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:35.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.943+0000 7f76fdc00700 1 -- 172.21.15.27:0/1098976610 shutdown_connections 2024-04-16T05:51:35.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.943+0000 7f76fdc00700 1 --2- 172.21.15.27:0/1098976610 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76f81002d0 0x7f76f8100740 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:35.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.943+0000 7f76fdc00700 1 --2- 172.21.15.27:0/1098976610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76f80ff020 0x7f76f80ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:35.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.943+0000 7f76fdc00700 1 -- 172.21.15.27:0/1098976610 >> 172.21.15.27:0/1098976610 conn(0x7f76f80fa950 msgr2=0x7f76f80fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:35.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.943+0000 7f76fdc00700 1 -- 172.21.15.27:0/1098976610 shutdown_connections 2024-04-16T05:51:35.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.943+0000 7f76fdc00700 1 -- 172.21.15.27:0/1098976610 wait complete. 2024-04-16T05:51:35.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.944+0000 7f76fdc00700 1 Processor -- start 2024-04-16T05:51:35.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.944+0000 7f76fdc00700 1 -- start start 2024-04-16T05:51:35.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.945+0000 7f76fdc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76f80ff020 0x7f76f810fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:35.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.945+0000 7f76fdc00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76f81002d0 0x7f76f81100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:35.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.945+0000 7f76fdc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f81106b0 con 0x7f76f80ff020 2024-04-16T05:51:35.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.945+0000 7f76fdc00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f8110820 con 0x7f76f81002d0 2024-04-16T05:51:35.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.945+0000 7f76f6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76f80ff020 0x7f76f810fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:35.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.945+0000 7f76f6a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76f80ff020 0x7f76f810fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48464/0 (socket says 172.21.15.27:48464) 2024-04-16T05:51:35.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.945+0000 7f76f6a00700 1 -- 172.21.15.27:0/417939234 learned_addr learned my addr 172.21.15.27:0/417939234 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:51:35.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.945+0000 7f76f6000700 1 --2- 172.21.15.27:0/417939234 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76f81002d0 0x7f76f81100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:35.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.945+0000 7f76f6a00700 1 -- 172.21.15.27:0/417939234 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76f81002d0 msgr2=0x7f76f81100b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:35.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.945+0000 7f76f6a00700 1 --2- 172.21.15.27:0/417939234 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76f81002d0 0x7f76f81100b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:35.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.945+0000 7f76f6a00700 1 -- 172.21.15.27:0/417939234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e4005040 con 0x7f76f80ff020 2024-04-16T05:51:35.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.946+0000 7f76f6a00700 1 --2- 172.21.15.27:0/417939234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76f80ff020 0x7f76f810fb70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f76e8002a50 tx=0x7f76e80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:35.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.946+0000 7f76eea00700 1 -- 172.21.15.27:0/417939234 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76e8007cc0 con 0x7f76f80ff020 2024-04-16T05:51:35.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.946+0000 7f76eea00700 1 -- 172.21.15.27:0/417939234 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f76e800f040 con 0x7f76f80ff020 2024-04-16T05:51:35.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.946+0000 7f76eea00700 1 -- 172.21.15.27:0/417939234 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76e800da30 con 0x7f76f80ff020 2024-04-16T05:51:35.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.946+0000 7f76fdc00700 1 -- 172.21.15.27:0/417939234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f8115280 con 0x7f76f80ff020 2024-04-16T05:51:35.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.946+0000 7f76fdc00700 1 -- 172.21.15.27:0/417939234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f8115840 con 0x7f76f80ff020 2024-04-16T05:51:35.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.947+0000 7f76fdc00700 1 -- 172.21.15.27:0/417939234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f804c710 con 0x7f76f80ff020 2024-04-16T05:51:35.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.949+0000 7f76eea00700 1 -- 172.21.15.27:0/417939234 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f76e800db90 con 0x7f76f80ff020 2024-04-16T05:51:35.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.949+0000 7f76eea00700 1 --2- 172.21.15.27:0/417939234 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f76dc06ec00 0x7f76dc0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:35.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.950+0000 7f76eea00700 1 -- 172.21.15.27:0/417939234 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6908+0+0 (secure 0 0 0) 0x7f76e808e790 con 0x7f76f80ff020 2024-04-16T05:51:35.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.950+0000 7f76f6000700 1 --2- 172.21.15.27:0/417939234 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f76dc06ec00 0x7f76dc0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:35.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.951+0000 7f76f6000700 1 --2- 172.21.15.27:0/417939234 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f76dc06ec00 0x7f76dc0710b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f76e400b3b0 tx=0x7f76e4009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:35.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:35.954+0000 7f76eea00700 1 -- 172.21.15.27:0/417939234 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f76e805aeb0 con 0x7f76f80ff020 2024-04-16T05:51:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.198+0000 7f76fdc00700 1 -- 172.21.15.27:0/417939234 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f76f81045f0 con 0x7f76dc06ec00 2024-04-16T05:51:36.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.229+0000 7f76eea00700 1 -- 172.21.15.27:0/417939234 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f76f81045f0 con 0x7f76dc06ec00 2024-04-16T05:51:36.228 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:51:36.228 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (7m) 34s ago 18m 13.5M - 0.25.0 c8568f914cd2 4ed63a8a6b33 2024-04-16T05:51:36.228 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (4m) 34s ago 18m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 4aa1f0006685 2024-04-16T05:51:36.229 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (4m) 5s ago 17m 6656k - 19.0.0-2884-g218a6914 a4e4811815e4 1c1912507d56 2024-04-16T05:51:36.229 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (7m) 34s ago 18m 74.0M - 9.4.7 954c08fa6188 f7ee2a1afd12 2024-04-16T05:51:36.229 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (41s) 34s ago 11m 11.0M - 19.0.0-2884-g218a6914 a4e4811815e4 b3c2178c537e 2024-04-16T05:51:36.229 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (72s) 34s ago 11m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 14b0c83f2891 2024-04-16T05:51:36.229 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (10s) 5s ago 11m 10.9M - 19.0.0-2884-g218a6914 a4e4811815e4 8ba95adfaaef 2024-04-16T05:51:36.229 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (11m) 5s ago 11m 17.3M - 17.2.7-847-g45317f95 b333581ccdf5 755a73c65a7f 2024-04-16T05:51:36.229 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:8443,9283,8765 running (7m) 34s ago 20m 509M - 19.0.0-2884-g218a6914 a4e4811815e4 b69a0b08e12b 2024-04-16T05:51:36.229 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283,8765 running (6m) 5s ago 17m 454M - 19.0.0-2884-g218a6914 a4e4811815e4 32e27bcd7e42 2024-04-16T05:51:36.229 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (6m) 34s ago 20m 53.4M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 299ca337e4ca 2024-04-16T05:51:36.229 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (5m) 5s ago 17m 39.4M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5db5f3037ed0 2024-04-16T05:51:36.229 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (8m) 34s ago 18m 7994k - 1.5.0 0da6a335fe13 5e29e29868b0 2024-04-16T05:51:36.229 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (7m) 5s ago 17m 8082k - 1.5.0 0da6a335fe13 a360922bc621 2024-04-16T05:51:36.229 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (4m) 34s ago 16m 45.9M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 cef03b67bc5a 2024-04-16T05:51:36.229 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (3m) 34s ago 15m 44.8M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3a6ab1fc6e79 2024-04-16T05:51:36.230 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (3m) 34s ago 14m 43.3M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3524731d7277 2024-04-16T05:51:36.230 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (2m) 5s ago 14m 44.3M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 897d3579f929 2024-04-16T05:51:36.230 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (2m) 5s ago 13m 42.0M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 fa97758ff7c5 2024-04-16T05:51:36.230 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (96s) 5s ago 13m 49.2M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 fd12260a820d 2024-04-16T05:51:36.230 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (6m) 34s ago 17m 59.2M - 2.43.0 a07b618ecd1d e70f55ba5ec0 2024-04-16T05:51:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.233+0000 7f76fdc00700 1 -- 172.21.15.27:0/417939234 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f76dc06ec00 msgr2=0x7f76dc0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.233+0000 7f76fdc00700 1 --2- 172.21.15.27:0/417939234 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f76dc06ec00 0x7f76dc0710b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f76e400b3b0 tx=0x7f76e4009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.233+0000 7f76fdc00700 1 -- 172.21.15.27:0/417939234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76f80ff020 msgr2=0x7f76f810fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.233+0000 7f76fdc00700 1 --2- 172.21.15.27:0/417939234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76f80ff020 0x7f76f810fb70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f76e8002a50 tx=0x7f76e80046b0 comp rx=0 tx=0).stop 2024-04-16T05:51:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.233+0000 7f76fdc00700 1 -- 172.21.15.27:0/417939234 shutdown_connections 2024-04-16T05:51:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.233+0000 7f76fdc00700 1 --2- 172.21.15.27:0/417939234 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f76dc06ec00 0x7f76dc0710b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.233+0000 7f76fdc00700 1 --2- 172.21.15.27:0/417939234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f76f80ff020 0x7f76f810fb70 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.233+0000 7f76fdc00700 1 --2- 172.21.15.27:0/417939234 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f76f81002d0 0x7f76f81100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.233+0000 7f76fdc00700 1 -- 172.21.15.27:0/417939234 >> 172.21.15.27:0/417939234 conn(0x7f76f80fa950 msgr2=0x7f76f80fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.233+0000 7f76fdc00700 1 -- 172.21.15.27:0/417939234 shutdown_connections 2024-04-16T05:51:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.233+0000 7f76fdc00700 1 -- 172.21.15.27:0/417939234 wait complete. 2024-04-16T05:51:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.390+0000 7f0b3fc00700 1 -- 172.21.15.27:0/1940294596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b380662f0 msgr2=0x7f0b380ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.390+0000 7f0b3fc00700 1 --2- 172.21.15.27:0/1940294596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b380662f0 0x7f0b380ffa30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c004940 tx=0x7f0b2c009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:36.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.390+0000 7f0b3fc00700 1 -- 172.21.15.27:0/1940294596 shutdown_connections 2024-04-16T05:51:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.390+0000 7f0b3fc00700 1 --2- 172.21.15.27:0/1940294596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b380fff70 0x7f0b38102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.390+0000 7f0b3fc00700 1 --2- 172.21.15.27:0/1940294596 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b380662f0 0x7f0b380ffa30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:36.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.390+0000 7f0b3fc00700 1 -- 172.21.15.27:0/1940294596 >> 172.21.15.27:0/1940294596 conn(0x7f0b380f74c0 msgr2=0x7f0b380f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:36.390 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:36 smithi027 ceph-mon[93635]: pgmap v224: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 B/s wr, 0 op/s 2024-04-16T05:51:36.390 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:36 smithi027 ceph-mon[93635]: from='client.34468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:51:36.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.390+0000 7f0b3fc00700 1 -- 172.21.15.27:0/1940294596 shutdown_connections 2024-04-16T05:51:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.390+0000 7f0b3fc00700 1 -- 172.21.15.27:0/1940294596 wait complete. 2024-04-16T05:51:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.391+0000 7f0b3fc00700 1 Processor -- start 2024-04-16T05:51:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.392+0000 7f0b3fc00700 1 -- start start 2024-04-16T05:51:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.392+0000 7f0b3fc00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b380662f0 0x7f0b3810fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.392+0000 7f0b3fc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b380fff70 0x7f0b38110260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.392+0000 7f0b3fc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b38110860 con 0x7f0b380fff70 2024-04-16T05:51:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.392+0000 7f0b3fc00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b38110fa0 con 0x7f0b380662f0 2024-04-16T05:51:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.392+0000 7f0b37e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b380fff70 0x7f0b38110260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.392+0000 7f0b37e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b380fff70 0x7f0b38110260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48484/0 (socket says 172.21.15.27:48484) 2024-04-16T05:51:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.392+0000 7f0b37e00700 1 -- 172.21.15.27:0/1782905135 learned_addr learned my addr 172.21.15.27:0/1782905135 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:51:36.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.392+0000 7f0b3d200700 1 --2- 172.21.15.27:0/1782905135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b380662f0 0x7f0b3810fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.393+0000 7f0b37e00700 1 -- 172.21.15.27:0/1782905135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b380662f0 msgr2=0x7f0b3810fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.393+0000 7f0b37e00700 1 --2- 172.21.15.27:0/1782905135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b380662f0 0x7f0b3810fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.393+0000 7f0b37e00700 1 -- 172.21.15.27:0/1782905135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c005040 con 0x7f0b380fff70 2024-04-16T05:51:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.393+0000 7f0b3d200700 1 --2- 172.21.15.27:0/1782905135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b380662f0 0x7f0b3810fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-16T05:51:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.393+0000 7f0b37e00700 1 --2- 172.21.15.27:0/1782905135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b380fff70 0x7f0b38110260 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0b24002a50 tx=0x7f0b240046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.394+0000 7f0b34c00700 1 -- 172.21.15.27:0/1782905135 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b24007cc0 con 0x7f0b380fff70 2024-04-16T05:51:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.394+0000 7f0b34c00700 1 -- 172.21.15.27:0/1782905135 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b2400f040 con 0x7f0b380fff70 2024-04-16T05:51:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.394+0000 7f0b34c00700 1 -- 172.21.15.27:0/1782905135 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b2400da70 con 0x7f0b380fff70 2024-04-16T05:51:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.394+0000 7f0b3fc00700 1 -- 172.21.15.27:0/1782905135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b38111280 con 0x7f0b380fff70 2024-04-16T05:51:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.394+0000 7f0b3fc00700 1 -- 172.21.15.27:0/1782905135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b381117f0 con 0x7f0b380fff70 2024-04-16T05:51:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.395+0000 7f0b3fc00700 1 -- 172.21.15.27:0/1782905135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b380f8ff0 con 0x7f0b380fff70 2024-04-16T05:51:36.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.397+0000 7f0b34c00700 1 -- 172.21.15.27:0/1782905135 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0b2400dc10 con 0x7f0b380fff70 2024-04-16T05:51:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.398+0000 7f0b34c00700 1 --2- 172.21.15.27:0/1782905135 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f0b2006ec10 0x7f0b200710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.398+0000 7f0b34c00700 1 -- 172.21.15.27:0/1782905135 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6908+0+0 (secure 0 0 0) 0x7f0b2408e150 con 0x7f0b380fff70 2024-04-16T05:51:36.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.399+0000 7f0b3d200700 1 --2- 172.21.15.27:0/1782905135 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f0b2006ec10 0x7f0b200710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:36.398 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.399+0000 7f0b3d200700 1 --2- 172.21.15.27:0/1782905135 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f0b2006ec10 0x7f0b200710c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00b3b0 tx=0x7f0b2c009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:36.402 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.403+0000 7f0b34c00700 1 -- 172.21.15.27:0/1782905135 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f0b2405a7f0 con 0x7f0b380fff70 2024-04-16T05:51:36.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:36 smithi122 ceph-mon[80930]: pgmap v224: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 B/s wr, 0 op/s 2024-04-16T05:51:36.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:36 smithi122 ceph-mon[80930]: from='client.34468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:51:36.719 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 530, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 105930 2024-04-16T05:51:36.719 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 530, success, outsize: 120 2024-04-16T05:51:36.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.724+0000 7f0b3fc00700 1 -- 172.21.15.27:0/1782905135 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0b3805f5f0 con 0x7f0b380fff70 2024-04-16T05:51:36.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.726+0000 7f0b34c00700 1 -- 172.21.15.27:0/1782905135 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+837 (secure 0 0 0) 0x7f0b2405a380 con 0x7f0b380fff70 2024-04-16T05:51:36.725 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:51:36.725 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:51:36.725 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:51:36.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:51:36.725 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:51:36.725 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:51:36.725 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:51:36.726 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:51:36.726 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T05:51:36.726 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:51:36.726 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:51:36.726 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 1, 2024-04-16T05:51:36.726 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 3 2024-04-16T05:51:36.726 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:51:36.726 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:51:36.726 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 17.2.7-847-g45317f95 (45317f95c5587af98fd8eaefff9013a4ba51d2f7) quincy (stable)": 1, 2024-04-16T05:51:36.726 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 13 2024-04-16T05:51:36.726 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:51:36.726 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:51:36.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.729+0000 7f0b3fc00700 1 -- 172.21.15.27:0/1782905135 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f0b2006ec10 msgr2=0x7f0b200710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:36.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.729+0000 7f0b3fc00700 1 --2- 172.21.15.27:0/1782905135 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f0b2006ec10 0x7f0b200710c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00b3b0 tx=0x7f0b2c009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:36.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.729+0000 7f0b3fc00700 1 -- 172.21.15.27:0/1782905135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b380fff70 msgr2=0x7f0b38110260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:36.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.729+0000 7f0b3fc00700 1 --2- 172.21.15.27:0/1782905135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b380fff70 0x7f0b38110260 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0b24002a50 tx=0x7f0b240046b0 comp rx=0 tx=0).stop 2024-04-16T05:51:36.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.729+0000 7f0b3fc00700 1 -- 172.21.15.27:0/1782905135 shutdown_connections 2024-04-16T05:51:36.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.729+0000 7f0b3fc00700 1 --2- 172.21.15.27:0/1782905135 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f0b2006ec10 0x7f0b200710c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:36.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.729+0000 7f0b3fc00700 1 --2- 172.21.15.27:0/1782905135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b380662f0 0x7f0b3810fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:36.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.729+0000 7f0b3fc00700 1 --2- 172.21.15.27:0/1782905135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b380fff70 0x7f0b38110260 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:36.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.729+0000 7f0b3fc00700 1 -- 172.21.15.27:0/1782905135 >> 172.21.15.27:0/1782905135 conn(0x7f0b380f74c0 msgr2=0x7f0b380fbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:36.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.729+0000 7f0b3fc00700 1 -- 172.21.15.27:0/1782905135 shutdown_connections 2024-04-16T05:51:36.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.730+0000 7f0b3fc00700 1 -- 172.21.15.27:0/1782905135 wait complete. 2024-04-16T05:51:36.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 532, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106018 2024-04-16T05:51:36.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 532, success, outsize: 120 2024-04-16T05:51:36.765 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 534, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106109 2024-04-16T05:51:36.765 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 534, success, outsize: 120 2024-04-16T05:51:36.792 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 536, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106206 2024-04-16T05:51:36.792 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 536, success, outsize: 120 2024-04-16T05:51:36.820 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 538, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106300 2024-04-16T05:51:36.820 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 538, success, outsize: 120 2024-04-16T05:51:36.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 540, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106388 2024-04-16T05:51:36.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 540, success, outsize: 120 2024-04-16T05:51:36.871 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 542, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106479 2024-04-16T05:51:36.871 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 542, success, outsize: 120 2024-04-16T05:51:36.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.881+0000 7fea71c00700 1 -- 172.21.15.27:0/3351469608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea6c1002d0 msgr2=0x7fea6c100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:36.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.881+0000 7fea71c00700 1 --2- 172.21.15.27:0/3351469608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea6c1002d0 0x7fea6c100740 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fea58004940 tx=0x7fea58009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:36.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.881+0000 7fea71c00700 1 -- 172.21.15.27:0/3351469608 shutdown_connections 2024-04-16T05:51:36.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.881+0000 7fea71c00700 1 --2- 172.21.15.27:0/3351469608 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea6c1002d0 0x7fea6c100740 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:36.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.881+0000 7fea71c00700 1 --2- 172.21.15.27:0/3351469608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea6c0ff020 0x7fea6c0ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:36.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.881+0000 7fea71c00700 1 -- 172.21.15.27:0/3351469608 >> 172.21.15.27:0/3351469608 conn(0x7fea6c0fa950 msgr2=0x7fea6c0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:36.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.881+0000 7fea71c00700 1 -- 172.21.15.27:0/3351469608 shutdown_connections 2024-04-16T05:51:36.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.881+0000 7fea71c00700 1 -- 172.21.15.27:0/3351469608 wait complete. 2024-04-16T05:51:36.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.882+0000 7fea71c00700 1 Processor -- start 2024-04-16T05:51:36.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.882+0000 7fea71c00700 1 -- start start 2024-04-16T05:51:36.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.882+0000 7fea71c00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea6c0ff020 0x7fea6c10fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:36.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.882+0000 7fea71c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea6c1002d0 0x7fea6c1100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:36.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.883+0000 7fea71c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c1106b0 con 0x7fea6c1002d0 2024-04-16T05:51:36.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.883+0000 7fea71c00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6c110820 con 0x7fea6c0ff020 2024-04-16T05:51:36.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.883+0000 7fea6a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea6c1002d0 0x7fea6c1100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:36.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.883+0000 7fea6a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea6c1002d0 0x7fea6c1100b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48500/0 (socket says 172.21.15.27:48500) 2024-04-16T05:51:36.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.883+0000 7fea6a000700 1 -- 172.21.15.27:0/2055744063 learned_addr learned my addr 172.21.15.27:0/2055744063 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:51:36.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.883+0000 7fea6aa00700 1 --2- 172.21.15.27:0/2055744063 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea6c0ff020 0x7fea6c10fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.883+0000 7fea6a000700 1 -- 172.21.15.27:0/2055744063 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea6c0ff020 msgr2=0x7fea6c10fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.883+0000 7fea6a000700 1 --2- 172.21.15.27:0/2055744063 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea6c0ff020 0x7fea6c10fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.883+0000 7fea6a000700 1 -- 172.21.15.27:0/2055744063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea58005040 con 0x7fea6c1002d0 2024-04-16T05:51:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.884+0000 7fea6a000700 1 --2- 172.21.15.27:0/2055744063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea6c1002d0 0x7fea6c1100b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fea5800b3b0 tx=0x7fea58009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.884+0000 7fea62a00700 1 -- 172.21.15.27:0/2055744063 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea5800e040 con 0x7fea6c1002d0 2024-04-16T05:51:36.883 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.884+0000 7fea62a00700 1 -- 172.21.15.27:0/2055744063 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea58004290 con 0x7fea6c1002d0 2024-04-16T05:51:36.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.884+0000 7fea62a00700 1 -- 172.21.15.27:0/2055744063 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea5801c3f0 con 0x7fea6c1002d0 2024-04-16T05:51:36.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.884+0000 7fea71c00700 1 -- 172.21.15.27:0/2055744063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea6c115220 con 0x7fea6c1002d0 2024-04-16T05:51:36.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.884+0000 7fea71c00700 1 -- 172.21.15.27:0/2055744063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea6c1157e0 con 0x7fea6c1002d0 2024-04-16T05:51:36.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.885+0000 7fea71c00700 1 -- 172.21.15.27:0/2055744063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea6c04c710 con 0x7fea6c1002d0 2024-04-16T05:51:36.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.886+0000 7fea62a00700 1 -- 172.21.15.27:0/2055744063 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fea5800bbf0 con 0x7fea6c1002d0 2024-04-16T05:51:36.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.887+0000 7fea62a00700 1 --2- 172.21.15.27:0/2055744063 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fea5006ec00 0x7fea500710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:36.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.888+0000 7fea62a00700 1 -- 172.21.15.27:0/2055744063 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6908+0+0 (secure 0 0 0) 0x7fea5808fd00 con 0x7fea6c1002d0 2024-04-16T05:51:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.888+0000 7fea6aa00700 1 --2- 172.21.15.27:0/2055744063 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fea5006ec00 0x7fea500710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:36.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.888+0000 7fea6aa00700 1 --2- 172.21.15.27:0/2055744063 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fea5006ec00 0x7fea500710b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fea5c007fd0 tx=0x7fea5c009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:36.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:36.892+0000 7fea62a00700 1 -- 172.21.15.27:0/2055744063 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fea5805c4a0 con 0x7fea6c1002d0 2024-04-16T05:51:36.900 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 544, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106576 2024-04-16T05:51:36.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 544, success, outsize: 120 2024-04-16T05:51:36.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 546, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106670 2024-04-16T05:51:36.929 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 546, success, outsize: 120 2024-04-16T05:51:36.957 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 548, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106758 2024-04-16T05:51:36.957 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 548, success, outsize: 120 2024-04-16T05:51:36.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 550, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106849 2024-04-16T05:51:36.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 550, success, outsize: 120 2024-04-16T05:51:37.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 552, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 106946 2024-04-16T05:51:37.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 552, success, outsize: 120 2024-04-16T05:51:37.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 554, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107040 2024-04-16T05:51:37.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 554, success, outsize: 120 2024-04-16T05:51:37.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 556, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107128 2024-04-16T05:51:37.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 556, success, outsize: 120 2024-04-16T05:51:37.089 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 558, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107219 2024-04-16T05:51:37.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 558, success, outsize: 120 2024-04-16T05:51:37.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 560, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107316 2024-04-16T05:51:37.116 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 560, success, outsize: 120 2024-04-16T05:51:37.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 562, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107410 2024-04-16T05:51:37.137 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 562, success, outsize: 120 2024-04-16T05:51:37.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 564, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107498 2024-04-16T05:51:37.157 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 564, success, outsize: 120 2024-04-16T05:51:37.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.165+0000 7fea71c00700 1 -- 172.21.15.27:0/2055744063 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fea6c107b10 con 0x7fea6c1002d0 2024-04-16T05:51:37.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.166+0000 7fea62a00700 1 -- 172.21.15.27:0/2055744063 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 30 v30) v1 ==== 76+0+1892 (secure 0 0 0) 0x7fea5808de60 con 0x7fea6c1002d0 2024-04-16T05:51:37.165 INFO:teuthology.orchestra.run.smithi027.stdout:e30 2024-04-16T05:51:37.165 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:51:37.165 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:51:37.165 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:51:37.165 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:51:37.165 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:51:37.165 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:51:37.165 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 29 2024-04-16T05:51:37.166 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:51:37.166 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:51:37.166 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:51:18.679503+0000 2024-04-16T05:51:37.166 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:51:37.166 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:51:37.166 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:51:37.166 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:51:37.166 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:51:37.166 INFO:teuthology.orchestra.run.smithi027.stdout:max_xattr_size 65536 2024-04-16T05:51:37.166 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:51:37.166 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:51:37.166 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 87 2024-04-16T05:51:37.166 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:51:37.166 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:51:37.166 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:51:37.167 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=24273} 2024-04-16T05:51:37.167 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:51:37.167 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:51:37.167 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:51:37.167 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:51:37.167 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:51:37.167 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:51:37.167 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:51:37.167 INFO:teuthology.orchestra.run.smithi027.stdout:bal_rank_mask -1 2024-04-16T05:51:37.167 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:51:37.167 INFO:teuthology.orchestra.run.smithi027.stdout:qdb_cluster leader: 24273 members: 24273 2024-04-16T05:51:37.167 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{0:24273} state up:active seq 165 join_fscid=1 addr [v2:172.21.15.122:6826/1078307975,v1:172.21.15.122:6827/1078307975] compat {c=[1],r=[1],i=[7ff]}] 2024-04-16T05:51:37.167 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:51:37.167 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:51:37.168 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:51:37.168 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:51:37.168 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{-1:34400} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6826/2831468256,v1:172.21.15.27:6827/2831468256] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:51:37.168 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{-1:34432} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6828/3219609959,v1:172.21.15.27:6829/3219609959] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:51:37.168 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:44337} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/97103507,v1:172.21.15.122:6825/97103507] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:51:37.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.169+0000 7fea71c00700 1 -- 172.21.15.27:0/2055744063 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fea5006ec00 msgr2=0x7fea500710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:37.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.169+0000 7fea71c00700 1 --2- 172.21.15.27:0/2055744063 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fea5006ec00 0x7fea500710b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fea5c007fd0 tx=0x7fea5c009040 comp rx=0 tx=0).stop 2024-04-16T05:51:37.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.169+0000 7fea71c00700 1 -- 172.21.15.27:0/2055744063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea6c1002d0 msgr2=0x7fea6c1100b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:37.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.169+0000 7fea71c00700 1 --2- 172.21.15.27:0/2055744063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea6c1002d0 0x7fea6c1100b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fea5800b3b0 tx=0x7fea58009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:37.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.169+0000 7fea71c00700 1 -- 172.21.15.27:0/2055744063 shutdown_connections 2024-04-16T05:51:37.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.169+0000 7fea71c00700 1 --2- 172.21.15.27:0/2055744063 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fea5006ec00 0x7fea500710b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:37.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.169+0000 7fea71c00700 1 --2- 172.21.15.27:0/2055744063 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea6c0ff020 0x7fea6c10fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:37.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.169+0000 7fea71c00700 1 --2- 172.21.15.27:0/2055744063 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea6c1002d0 0x7fea6c1100b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:37.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.169+0000 7fea71c00700 1 -- 172.21.15.27:0/2055744063 >> 172.21.15.27:0/2055744063 conn(0x7fea6c0fa950 msgr2=0x7fea6c0fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:37.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.169+0000 7fea71c00700 1 -- 172.21.15.27:0/2055744063 shutdown_connections 2024-04-16T05:51:37.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.169+0000 7fea71c00700 1 -- 172.21.15.27:0/2055744063 wait complete. 2024-04-16T05:51:37.170 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 30 2024-04-16T05:51:37.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 566, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107589 2024-04-16T05:51:37.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 566, success, outsize: 120 2024-04-16T05:51:37.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 568, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107686 2024-04-16T05:51:37.203 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 568, success, outsize: 120 2024-04-16T05:51:37.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 570, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107780 2024-04-16T05:51:37.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 570, success, outsize: 120 2024-04-16T05:51:37.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 572, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107868 2024-04-16T05:51:37.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 572, success, outsize: 120 2024-04-16T05:51:37.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 574, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 107959 2024-04-16T05:51:37.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 574, success, outsize: 120 2024-04-16T05:51:37.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 576, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108056 2024-04-16T05:51:37.292 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 576, success, outsize: 120 2024-04-16T05:51:37.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 578, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108150 2024-04-16T05:51:37.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 578, success, outsize: 120 2024-04-16T05:51:37.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.323+0000 7f1d09600700 1 -- 172.21.15.27:0/2373262959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d040fcdc0 msgr2=0x7f1d040fd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:37.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.323+0000 7f1d09600700 1 --2- 172.21.15.27:0/2373262959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d040fcdc0 0x7f1d040fd1d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4004940 tx=0x7f1cf4009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:37.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.323+0000 7f1d09600700 1 -- 172.21.15.27:0/2373262959 shutdown_connections 2024-04-16T05:51:37.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.323+0000 7f1d09600700 1 --2- 172.21.15.27:0/2373262959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d040fd710 0x7f1d040fe070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:37.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.323+0000 7f1d09600700 1 --2- 172.21.15.27:0/2373262959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d040fcdc0 0x7f1d040fd1d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:37.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.323+0000 7f1d09600700 1 -- 172.21.15.27:0/2373262959 >> 172.21.15.27:0/2373262959 conn(0x7f1d040f8540 msgr2=0x7f1d040fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:37.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.324+0000 7f1d09600700 1 -- 172.21.15.27:0/2373262959 shutdown_connections 2024-04-16T05:51:37.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.324+0000 7f1d09600700 1 -- 172.21.15.27:0/2373262959 wait complete. 2024-04-16T05:51:37.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.325+0000 7f1d09600700 1 Processor -- start 2024-04-16T05:51:37.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.325+0000 7f1d09600700 1 -- start start 2024-04-16T05:51:37.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.325+0000 7f1d09600700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d040fcdc0 0x7f1d0410fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:37.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.325+0000 7f1d09600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d040fd710 0x7f1d041102a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:37.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.325+0000 7f1d09600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d041108a0 con 0x7f1d040fd710 2024-04-16T05:51:37.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.325+0000 7f1d09600700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d04110fe0 con 0x7f1d040fcdc0 2024-04-16T05:51:37.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.326+0000 7f1d02000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d040fd710 0x7f1d041102a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:37.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.326+0000 7f1d02000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d040fd710 0x7f1d041102a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48518/0 (socket says 172.21.15.27:48518) 2024-04-16T05:51:37.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.326+0000 7f1d02000700 1 -- 172.21.15.27:0/499675885 learned_addr learned my addr 172.21.15.27:0/499675885 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:51:37.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.326+0000 7f1d02a00700 1 --2- 172.21.15.27:0/499675885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d040fcdc0 0x7f1d0410fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:37.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.326+0000 7f1d02000700 1 -- 172.21.15.27:0/499675885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d040fcdc0 msgr2=0x7f1d0410fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:37.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.326+0000 7f1d02000700 1 --2- 172.21.15.27:0/499675885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d040fcdc0 0x7f1d0410fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:37.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.326+0000 7f1d02000700 1 -- 172.21.15.27:0/499675885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf4005040 con 0x7f1d040fd710 2024-04-16T05:51:37.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.327+0000 7f1d02000700 1 --2- 172.21.15.27:0/499675885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d040fd710 0x7f1d041102a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0002a80 tx=0x7f1cf0004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:37.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.327+0000 7f1cfaa00700 1 -- 172.21.15.27:0/499675885 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cf0004e80 con 0x7f1d040fd710 2024-04-16T05:51:37.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.327+0000 7f1cfaa00700 1 -- 172.21.15.27:0/499675885 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1cf000f040 con 0x7f1d040fd710 2024-04-16T05:51:37.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.327+0000 7f1d09600700 1 -- 172.21.15.27:0/499675885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d041112c0 con 0x7f1d040fd710 2024-04-16T05:51:37.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.327+0000 7f1cfaa00700 1 -- 172.21.15.27:0/499675885 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cf000daa0 con 0x7f1d040fd710 2024-04-16T05:51:37.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.327+0000 7f1d09600700 1 -- 172.21.15.27:0/499675885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d04111880 con 0x7f1d040fd710 2024-04-16T05:51:37.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.328+0000 7f1d09600700 1 -- 172.21.15.27:0/499675885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce40052e0 con 0x7f1d040fd710 2024-04-16T05:51:37.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.330+0000 7f1cfaa00700 1 -- 172.21.15.27:0/499675885 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1cf0007cc0 con 0x7f1d040fd710 2024-04-16T05:51:37.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.330+0000 7f1cfaa00700 1 --2- 172.21.15.27:0/499675885 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1ce806ec00 0x7f1ce80710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:37.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.331+0000 7f1cfaa00700 1 -- 172.21.15.27:0/499675885 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6908+0+0 (secure 0 0 0) 0x7f1cf008df00 con 0x7f1d040fd710 2024-04-16T05:51:37.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.331+0000 7f1d02a00700 1 --2- 172.21.15.27:0/499675885 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1ce806ec00 0x7f1ce80710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:37.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.331+0000 7f1d02a00700 1 --2- 172.21.15.27:0/499675885 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1ce806ec00 0x7f1ce80710b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4004890 tx=0x7f1cf4009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:37.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 580, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108238 2024-04-16T05:51:37.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 580, success, outsize: 120 2024-04-16T05:51:37.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.335+0000 7f1cfaa00700 1 -- 172.21.15.27:0/499675885 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1cf005a6c0 con 0x7f1d040fd710 2024-04-16T05:51:37.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 582, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108329 2024-04-16T05:51:37.351 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 582, success, outsize: 120 2024-04-16T05:51:37.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 584, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108426 2024-04-16T05:51:37.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 584, success, outsize: 120 2024-04-16T05:51:37.376 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:37 smithi122 ceph-mon[80930]: from='client.34472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:51:37.376 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:37 smithi122 ceph-mon[80930]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:51:37.376 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:37 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1782905135' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:51:37.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:37 smithi027 ceph-mon[93635]: from='client.34472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:51:37.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:37 smithi027 ceph-mon[93635]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:51:37.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:37 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1782905135' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:51:37.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.583+0000 7f1d09600700 1 -- 172.21.15.27:0/499675885 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ce4000bf0 con 0x7f1ce806ec00 2024-04-16T05:51:37.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.586+0000 7f1cfaa00700 1 -- 172.21.15.27:0/499675885 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+416 (secure 0 0 0) 0x7f1ce4000bf0 con 0x7f1ce806ec00 2024-04-16T05:51:37.585 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:51:37.586 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T05:51:37.586 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:51:37.586 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:51:37.586 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [ 2024-04-16T05:51:37.586 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr", 2024-04-16T05:51:37.586 INFO:teuthology.orchestra.run.smithi027.stdout: "mon", 2024-04-16T05:51:37.586 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-16T05:51:37.586 INFO:teuthology.orchestra.run.smithi027.stdout: "osd" 2024-04-16T05:51:37.586 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-16T05:51:37.586 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "15/21 daemons upgraded", 2024-04-16T05:51:37.586 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Currently upgrading mds daemons", 2024-04-16T05:51:37.586 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:51:37.586 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:51:37.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.589+0000 7f1d09600700 1 -- 172.21.15.27:0/499675885 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1ce806ec00 msgr2=0x7f1ce80710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:37.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.589+0000 7f1d09600700 1 --2- 172.21.15.27:0/499675885 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1ce806ec00 0x7f1ce80710b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4004890 tx=0x7f1cf4009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:37.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.589+0000 7f1d09600700 1 -- 172.21.15.27:0/499675885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d040fd710 msgr2=0x7f1d041102a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:37.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.589+0000 7f1d09600700 1 --2- 172.21.15.27:0/499675885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d040fd710 0x7f1d041102a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0002a80 tx=0x7f1cf0004630 comp rx=0 tx=0).stop 2024-04-16T05:51:37.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.589+0000 7f1d09600700 1 -- 172.21.15.27:0/499675885 shutdown_connections 2024-04-16T05:51:37.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.589+0000 7f1d09600700 1 --2- 172.21.15.27:0/499675885 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1ce806ec00 0x7f1ce80710b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:37.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.589+0000 7f1d09600700 1 --2- 172.21.15.27:0/499675885 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d040fcdc0 0x7f1d0410fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:37.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.589+0000 7f1d09600700 1 --2- 172.21.15.27:0/499675885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d040fd710 0x7f1d041102a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:37.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.589+0000 7f1d09600700 1 -- 172.21.15.27:0/499675885 >> 172.21.15.27:0/499675885 conn(0x7f1d040f8540 msgr2=0x7f1d040f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:37.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.589+0000 7f1d09600700 1 -- 172.21.15.27:0/499675885 shutdown_connections 2024-04-16T05:51:37.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.589+0000 7f1d09600700 1 -- 172.21.15.27:0/499675885 wait complete. 2024-04-16T05:51:37.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.741+0000 7fc357600700 1 -- 172.21.15.27:0/440897245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3500662f0 msgr2=0x7fc3500ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:37.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.741+0000 7fc357600700 1 --2- 172.21.15.27:0/440897245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3500662f0 0x7fc3500ffa30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc340004940 tx=0x7fc340009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:37.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.741+0000 7fc357600700 1 -- 172.21.15.27:0/440897245 shutdown_connections 2024-04-16T05:51:37.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.741+0000 7fc357600700 1 --2- 172.21.15.27:0/440897245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3500fff70 0x7fc350102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:37.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.741+0000 7fc357600700 1 --2- 172.21.15.27:0/440897245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3500662f0 0x7fc3500ffa30 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:37.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.741+0000 7fc357600700 1 -- 172.21.15.27:0/440897245 >> 172.21.15.27:0/440897245 conn(0x7fc3500f74c0 msgr2=0x7fc3500f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:37.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.741+0000 7fc357600700 1 -- 172.21.15.27:0/440897245 shutdown_connections 2024-04-16T05:51:37.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.742+0000 7fc357600700 1 -- 172.21.15.27:0/440897245 wait complete. 2024-04-16T05:51:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.742+0000 7fc357600700 1 Processor -- start 2024-04-16T05:51:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.743+0000 7fc357600700 1 -- start start 2024-04-16T05:51:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.743+0000 7fc357600700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3500662f0 0x7fc350107120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.743+0000 7fc357600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3500fff70 0x7fc350107660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.743+0000 7fc357600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc350107d40 con 0x7fc3500fff70 2024-04-16T05:51:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.743+0000 7fc357600700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3501084b0 con 0x7fc3500662f0 2024-04-16T05:51:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.743+0000 7fc347e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3500fff70 0x7fc350107660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.743+0000 7fc347e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3500fff70 0x7fc350107660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48540/0 (socket says 172.21.15.27:48540) 2024-04-16T05:51:37.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.743+0000 7fc347e00700 1 -- 172.21.15.27:0/755735699 learned_addr learned my addr 172.21.15.27:0/755735699 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:51:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.744+0000 7fc354c00700 1 --2- 172.21.15.27:0/755735699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3500662f0 0x7fc350107120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.744+0000 7fc347e00700 1 -- 172.21.15.27:0/755735699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3500662f0 msgr2=0x7fc350107120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:37.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.744+0000 7fc347e00700 1 --2- 172.21.15.27:0/755735699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3500662f0 0x7fc350107120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.744+0000 7fc347e00700 1 -- 172.21.15.27:0/755735699 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc340005040 con 0x7fc3500fff70 2024-04-16T05:51:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.744+0000 7fc347e00700 1 --2- 172.21.15.27:0/755735699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3500fff70 0x7fc350107660 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc33800b790 tx=0x7fc338009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.744+0000 7fc34d600700 1 -- 172.21.15.27:0/755735699 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc33800f770 con 0x7fc3500fff70 2024-04-16T05:51:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.744+0000 7fc34d600700 1 -- 172.21.15.27:0/755735699 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc33800fda0 con 0x7fc3500fff70 2024-04-16T05:51:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.744+0000 7fc357600700 1 -- 172.21.15.27:0/755735699 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc350108790 con 0x7fc3500fff70 2024-04-16T05:51:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.744+0000 7fc34d600700 1 -- 172.21.15.27:0/755735699 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc33800d310 con 0x7fc3500fff70 2024-04-16T05:51:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.744+0000 7fc357600700 1 -- 172.21.15.27:0/755735699 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc350108d50 con 0x7fc3500fff70 2024-04-16T05:51:37.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.746+0000 7fc357600700 1 -- 172.21.15.27:0/755735699 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc35005f5f0 con 0x7fc3500fff70 2024-04-16T05:51:37.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.747+0000 7fc34d600700 1 -- 172.21.15.27:0/755735699 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc338010420 con 0x7fc3500fff70 2024-04-16T05:51:37.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.748+0000 7fc34d600700 1 --2- 172.21.15.27:0/755735699 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc33406ec10 0x7fc3340710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:51:37.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.748+0000 7fc34d600700 1 -- 172.21.15.27:0/755735699 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 6908+0+0 (secure 0 0 0) 0x7fc33808e9f0 con 0x7fc3500fff70 2024-04-16T05:51:37.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.748+0000 7fc354c00700 1 --2- 172.21.15.27:0/755735699 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc33406ec10 0x7fc3340710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:51:37.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.749+0000 7fc354c00700 1 --2- 172.21.15.27:0/755735699 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc33406ec10 0x7fc3340710c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc34000b3b0 tx=0x7fc340009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:51:37.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:37.752+0000 7fc34d600700 1 -- 172.21.15.27:0/755735699 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc3380576a0 con 0x7fc3500fff70 2024-04-16T05:51:38.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:38.060+0000 7fc357600700 1 -- 172.21.15.27:0/755735699 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fc35004bed0 con 0x7fc3500fff70 2024-04-16T05:51:38.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:38.061+0000 7fc34d600700 1 -- 172.21.15.27:0/755735699 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fc33805ad60 con 0x7fc3500fff70 2024-04-16T05:51:38.060 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:51:38.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:38.064+0000 7fc357600700 1 -- 172.21.15.27:0/755735699 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc33406ec10 msgr2=0x7fc3340710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:38.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:38.064+0000 7fc357600700 1 --2- 172.21.15.27:0/755735699 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc33406ec10 0x7fc3340710c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc34000b3b0 tx=0x7fc340009e30 comp rx=0 tx=0).stop 2024-04-16T05:51:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:38.064+0000 7fc357600700 1 -- 172.21.15.27:0/755735699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3500fff70 msgr2=0x7fc350107660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:51:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:38.064+0000 7fc357600700 1 --2- 172.21.15.27:0/755735699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3500fff70 0x7fc350107660 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc33800b790 tx=0x7fc338009250 comp rx=0 tx=0).stop 2024-04-16T05:51:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:38.064+0000 7fc357600700 1 -- 172.21.15.27:0/755735699 shutdown_connections 2024-04-16T05:51:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:38.064+0000 7fc357600700 1 --2- 172.21.15.27:0/755735699 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc33406ec10 0x7fc3340710c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:38.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:38.064+0000 7fc357600700 1 --2- 172.21.15.27:0/755735699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3500662f0 0x7fc350107120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:38.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:38.064+0000 7fc357600700 1 --2- 172.21.15.27:0/755735699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc3500fff70 0x7fc350107660 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:51:38.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:38.064+0000 7fc357600700 1 -- 172.21.15.27:0/755735699 >> 172.21.15.27:0/755735699 conn(0x7fc3500f74c0 msgr2=0x7fc3500fbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:51:38.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:38.064+0000 7fc357600700 1 -- 172.21.15.27:0/755735699 shutdown_connections 2024-04-16T05:51:38.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:51:38.064+0000 7fc357600700 1 -- 172.21.15.27:0/755735699 wait complete. 2024-04-16T05:51:38.436 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:38 smithi027 ceph-mon[93635]: pgmap v225: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:51:38.436 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:38 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/2055744063' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:51:38.436 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:38 smithi027 ceph-mon[93635]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:51:38.436 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:38 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/755735699' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:51:38.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:38 smithi122 ceph-mon[80930]: pgmap v225: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:51:38.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:38 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/2055744063' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:51:38.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:38 smithi122 ceph-mon[80930]: from='client.34488 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:51:38.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:38 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/755735699' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:51:39.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:39 smithi027 ceph-mon[93635]: Detected new or changed devices on smithi122 2024-04-16T05:51:39.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:39 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:39.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:39 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:39.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:39 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:51:39.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:39 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:51:39.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:39 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:39.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:39 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:51:39.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:39 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:51:39.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:39 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:51:39.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:39 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:51:39.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:39 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:51:39.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:39 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi122.rulewi"]}]: dispatch 2024-04-16T05:51:39.638 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:39 smithi122 ceph-mon[80930]: Detected new or changed devices on smithi122 2024-04-16T05:51:39.638 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:39 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:39.639 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:39 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:39.639 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:39 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:51:39.639 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:39 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:51:39.639 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:39 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:39.639 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:39 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:51:39.639 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:39 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:51:39.639 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:39 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:51:39.639 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:39 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:51:39.639 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:39 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:51:39.639 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:39 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds ok-to-stop", "ids": ["cephfs.smithi122.rulewi"]}]: dispatch 2024-04-16T05:51:40.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:40 smithi122 ceph-mon[80930]: Upgrade: It appears safe to stop mds.cephfs.smithi122.rulewi 2024-04-16T05:51:40.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:40 smithi122 ceph-mon[80930]: pgmap v226: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:51:40.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:40 smithi027 ceph-mon[93635]: Upgrade: It appears safe to stop mds.cephfs.smithi122.rulewi 2024-04-16T05:51:40.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:40 smithi027 ceph-mon[93635]: pgmap v226: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:51:42.151 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:42 smithi122 ceph-mon[80930]: Upgrade: Updating mds.cephfs.smithi122.rulewi 2024-04-16T05:51:42.151 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:42 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:42.151 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:42 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.rulewi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:51:42.151 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:42 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:51:42.152 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:42 smithi122 ceph-mon[80930]: Deploying daemon mds.cephfs.smithi122.rulewi on smithi122 2024-04-16T05:51:42.152 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:42 smithi122 ceph-mon[80930]: pgmap v227: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:51:42.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:42 smithi027 ceph-mon[93635]: Upgrade: Updating mds.cephfs.smithi122.rulewi 2024-04-16T05:51:42.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:42 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:42.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:42 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi122.rulewi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-16T05:51:42.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:42 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:51:42.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:42 smithi027 ceph-mon[93635]: Deploying daemon mds.cephfs.smithi122.rulewi on smithi122 2024-04-16T05:51:42.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:42 smithi027 ceph-mon[93635]: pgmap v227: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:51:44.032 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 474, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 128060 2024-04-16T05:51:44.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 474, success, outsize: 120 2024-04-16T05:51:44.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 586, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108824 2024-04-16T05:51:44.033 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 586, success, outsize: 120 2024-04-16T05:51:44.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:44 smithi122 ceph-mon[80930]: pgmap v228: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:51:44.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:44 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:51:44.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:44 smithi122 ceph-mon[80930]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T05:51:44.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:44 smithi122 ceph-mon[80930]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T05:51:44.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:44 smithi122 ceph-mon[80930]: osdmap e88: 6 total, 6 up, 6 in 2024-04-16T05:51:44.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:44 smithi122 ceph-mon[80930]: Standby daemon mds.cephfs.smithi027.hkhpuq assigned to filesystem cephfs as rank 0 2024-04-16T05:51:44.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:44 smithi122 ceph-mon[80930]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T05:51:44.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:44 smithi122 ceph-mon[80930]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-16T05:51:44.272 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:44 smithi122 ceph-mon[80930]: fsmap cephfs:1/1 {0=cephfs.smithi027.hkhpuq=up:replay} 2 up:standby 2024-04-16T05:51:44.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:44 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027[93631]: 2024-04-16T05:51:44.010+0000 7fdd83600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T05:51:44.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:44 smithi027 ceph-mon[93635]: pgmap v228: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:51:44.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:51:44.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:44 smithi027 ceph-mon[93635]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-16T05:51:44.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:44 smithi027 ceph-mon[93635]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-16T05:51:44.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:44 smithi027 ceph-mon[93635]: osdmap e88: 6 total, 6 up, 6 in 2024-04-16T05:51:44.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:44 smithi027 ceph-mon[93635]: Standby daemon mds.cephfs.smithi027.hkhpuq assigned to filesystem cephfs as rank 0 2024-04-16T05:51:44.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:44 smithi027 ceph-mon[93635]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-16T05:51:44.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:44 smithi027 ceph-mon[93635]: fsmap cephfs:0/1 3 up:standby, 1 failed 2024-04-16T05:51:44.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:44 smithi027 ceph-mon[93635]: fsmap cephfs:1/1 {0=cephfs.smithi027.hkhpuq=up:replay} 2 up:standby 2024-04-16T05:51:45.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 588, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 108829 2024-04-16T05:51:45.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 588, success, outsize: 120 2024-04-16T05:51:45.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 476, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 128072 2024-04-16T05:51:45.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 476, success, outsize: 120 2024-04-16T05:51:45.210 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:45 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.rulewi"}]: dispatch 2024-04-16T05:51:45.210 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:45 smithi122 ceph-mon[80930]: mds.? [v2:172.21.15.27:6826/2831468256,v1:172.21.15.27:6827/2831468256] up:reconnect 2024-04-16T05:51:45.210 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:45 smithi122 ceph-mon[80930]: fsmap cephfs:1/1 {0=cephfs.smithi027.hkhpuq=up:reconnect} 2 up:standby 2024-04-16T05:51:45.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:45 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.rulewi"}]: dispatch 2024-04-16T05:51:45.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:45 smithi027 ceph-mon[93635]: mds.? [v2:172.21.15.27:6826/2831468256,v1:172.21.15.27:6827/2831468256] up:reconnect 2024-04-16T05:51:45.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:45 smithi027 ceph-mon[93635]: fsmap cephfs:1/1 {0=cephfs.smithi027.hkhpuq=up:reconnect} 2 up:standby 2024-04-16T05:51:46.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:46 smithi122 ceph-mon[80930]: reconnect by client.24301 192.168.0.1:0/790915989 after 0.00299992 2024-04-16T05:51:46.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:46 smithi122 ceph-mon[80930]: reconnect by client.14518 192.168.0.1:0/676551963 after 0.00499987 2024-04-16T05:51:46.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:46 smithi122 ceph-mon[80930]: pgmap v230: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:51:46.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:46 smithi122 ceph-mon[80930]: mds.? [v2:172.21.15.27:6826/2831468256,v1:172.21.15.27:6827/2831468256] up:rejoin 2024-04-16T05:51:46.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:46 smithi122 ceph-mon[80930]: fsmap cephfs:1/1 {0=cephfs.smithi027.hkhpuq=up:rejoin} 2 up:standby 2024-04-16T05:51:46.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:46 smithi122 ceph-mon[80930]: daemon mds.cephfs.smithi027.hkhpuq is now active in filesystem cephfs as rank 0 2024-04-16T05:51:46.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:46 smithi027 ceph-mon[93635]: reconnect by client.24301 192.168.0.1:0/790915989 after 0.00299992 2024-04-16T05:51:46.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:46 smithi027 ceph-mon[93635]: reconnect by client.14518 192.168.0.1:0/676551963 after 0.00499987 2024-04-16T05:51:46.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:46 smithi027 ceph-mon[93635]: pgmap v230: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:51:46.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:46 smithi027 ceph-mon[93635]: mds.? [v2:172.21.15.27:6826/2831468256,v1:172.21.15.27:6827/2831468256] up:rejoin 2024-04-16T05:51:46.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:46 smithi027 ceph-mon[93635]: fsmap cephfs:1/1 {0=cephfs.smithi027.hkhpuq=up:rejoin} 2 up:standby 2024-04-16T05:51:46.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:46 smithi027 ceph-mon[93635]: daemon mds.cephfs.smithi027.hkhpuq is now active in filesystem cephfs as rank 0 2024-04-16T05:51:47.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:47 smithi122 ceph-mon[80930]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T05:51:47.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:47 smithi122 ceph-mon[80930]: Cluster is now healthy 2024-04-16T05:51:47.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:47 smithi122 ceph-mon[80930]: mds.? [v2:172.21.15.27:6826/2831468256,v1:172.21.15.27:6827/2831468256] up:active 2024-04-16T05:51:47.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:47 smithi122 ceph-mon[80930]: fsmap cephfs:1 {0=cephfs.smithi027.hkhpuq=up:active} 2 up:standby 2024-04-16T05:51:47.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:47 smithi027 ceph-mon[93635]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-16T05:51:47.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:47 smithi027 ceph-mon[93635]: Cluster is now healthy 2024-04-16T05:51:47.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:47 smithi027 ceph-mon[93635]: mds.? [v2:172.21.15.27:6826/2831468256,v1:172.21.15.27:6827/2831468256] up:active 2024-04-16T05:51:47.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:47 smithi027 ceph-mon[93635]: fsmap cephfs:1 {0=cephfs.smithi027.hkhpuq=up:active} 2 up:standby 2024-04-16T05:51:48.438 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:48 smithi027 ceph-mon[93635]: pgmap v231: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 0 op/s 2024-04-16T05:51:48.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:48 smithi122 ceph-mon[80930]: pgmap v231: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 204 B/s wr, 0 op/s 2024-04-16T05:51:50.267 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:50 smithi122 ceph-mon[80930]: pgmap v232: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 3.0 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-16T05:51:50.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:50 smithi027 ceph-mon[93635]: pgmap v232: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 3.0 KiB/s rd, 204 B/s wr, 2 op/s 2024-04-16T05:51:52.460 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:52 smithi122 ceph-mon[80930]: pgmap v233: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 6.0 KiB/s rd, 204 B/s wr, 5 op/s 2024-04-16T05:51:52.460 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:52 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:52.461 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:52 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:52.461 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:52 smithi122 ceph-mon[80930]: mds.? [v2:172.21.15.122:6826/1030624816,v1:172.21.15.122:6827/1030624816] up:boot 2024-04-16T05:51:52.461 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:52 smithi122 ceph-mon[80930]: fsmap cephfs:1 {0=cephfs.smithi027.hkhpuq=up:active} 3 up:standby 2024-04-16T05:51:52.461 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:52 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.rulewi"}]: dispatch 2024-04-16T05:51:52.461 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:52 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:51:52.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:52 smithi027 ceph-mon[93635]: pgmap v233: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 6.0 KiB/s rd, 204 B/s wr, 5 op/s 2024-04-16T05:51:52.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:52 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:52.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:52 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:52.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:52 smithi027 ceph-mon[93635]: mds.? [v2:172.21.15.122:6826/1030624816,v1:172.21.15.122:6827/1030624816] up:boot 2024-04-16T05:51:52.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:52 smithi027 ceph-mon[93635]: fsmap cephfs:1 {0=cephfs.smithi027.hkhpuq=up:active} 3 up:standby 2024-04-16T05:51:52.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:52 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi122.rulewi"}]: dispatch 2024-04-16T05:51:52.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:52 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:51:54.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:54 smithi122 ceph-mon[80930]: pgmap v234: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 8.7 KiB/s rd, 204 B/s wr, 8 op/s 2024-04-16T05:51:54.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:54 smithi027 ceph-mon[93635]: pgmap v234: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 8.7 KiB/s rd, 204 B/s wr, 8 op/s 2024-04-16T05:51:56.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:55 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:56.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:55 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:56.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:55 smithi122 ceph-mon[80930]: pgmap v235: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 7.9 KiB/s rd, 185 B/s wr, 7 op/s 2024-04-16T05:51:56.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:55 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:56.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:55 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:56.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:55 smithi027 ceph-mon[93635]: pgmap v235: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 7.9 KiB/s rd, 185 B/s wr, 7 op/s 2024-04-16T05:51:58.148 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:57 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:58.148 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:57 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:58.148 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:57 smithi122 ceph-mon[80930]: pgmap v236: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 7.2 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-16T05:51:58.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:57 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:58.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:57 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:51:58.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:57 smithi027 ceph-mon[93635]: pgmap v236: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 7.2 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-16T05:51:59.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:51:59 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:51:59.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:51:59 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:52:00.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:00 smithi122 ceph-mon[80930]: pgmap v237: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-16T05:52:00.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:00 smithi027 ceph-mon[93635]: pgmap v237: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-16T05:52:00.752 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 590, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109823 2024-04-16T05:52:00.753 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 590, success, outsize: 120 2024-04-16T05:52:00.783 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 592, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 109911 2024-04-16T05:52:00.783 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 592, success, outsize: 120 2024-04-16T05:52:00.808 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 594, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110002 2024-04-16T05:52:00.809 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 594, success, outsize: 120 2024-04-16T05:52:00.837 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 596, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110099 2024-04-16T05:52:00.837 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 596, success, outsize: 120 2024-04-16T05:52:00.867 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 598, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110193 2024-04-16T05:52:00.867 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 598, success, outsize: 120 2024-04-16T05:52:00.895 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 600, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110281 2024-04-16T05:52:00.895 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 600, success, outsize: 120 2024-04-16T05:52:00.916 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 602, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110372 2024-04-16T05:52:00.917 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 602, success, outsize: 120 2024-04-16T05:52:00.939 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 604, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110469 2024-04-16T05:52:00.940 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 604, success, outsize: 120 2024-04-16T05:52:00.962 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 606, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110563 2024-04-16T05:52:00.962 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 606, success, outsize: 120 2024-04-16T05:52:00.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 608, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110651 2024-04-16T05:52:00.985 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 608, success, outsize: 120 2024-04-16T05:52:01.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 610, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110742 2024-04-16T05:52:01.009 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 610, success, outsize: 120 2024-04-16T05:52:01.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 612, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110839 2024-04-16T05:52:01.034 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 612, success, outsize: 120 2024-04-16T05:52:01.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 614, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 110933 2024-04-16T05:52:01.058 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 614, success, outsize: 120 2024-04-16T05:52:01.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 616, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111021 2024-04-16T05:52:01.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 616, success, outsize: 120 2024-04-16T05:52:01.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 618, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111112 2024-04-16T05:52:01.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 618, success, outsize: 120 2024-04-16T05:52:01.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 620, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111209 2024-04-16T05:52:01.127 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 620, success, outsize: 120 2024-04-16T05:52:01.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 622, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111303 2024-04-16T05:52:01.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 622, success, outsize: 120 2024-04-16T05:52:01.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 624, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111391 2024-04-16T05:52:01.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 624, success, outsize: 120 2024-04-16T05:52:01.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 626, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111482 2024-04-16T05:52:01.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 626, success, outsize: 120 2024-04-16T05:52:01.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 628, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111579 2024-04-16T05:52:01.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 628, success, outsize: 120 2024-04-16T05:52:01.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 630, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111673 2024-04-16T05:52:01.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 630, success, outsize: 120 2024-04-16T05:52:01.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 632, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111761 2024-04-16T05:52:01.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 632, success, outsize: 120 2024-04-16T05:52:01.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 634, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111852 2024-04-16T05:52:01.260 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 634, success, outsize: 120 2024-04-16T05:52:01.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 636, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 111949 2024-04-16T05:52:01.281 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 636, success, outsize: 120 2024-04-16T05:52:01.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 638, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112043 2024-04-16T05:52:01.302 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 638, success, outsize: 120 2024-04-16T05:52:01.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 640, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112131 2024-04-16T05:52:01.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 640, success, outsize: 120 2024-04-16T05:52:01.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 642, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112222 2024-04-16T05:52:01.340 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 642, success, outsize: 120 2024-04-16T05:52:01.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 644, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 112319 2024-04-16T05:52:01.361 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 644, success, outsize: 120 2024-04-16T05:52:02.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:02 smithi122 ceph-mon[80930]: pgmap v238: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 4.6 KiB/s rd, 4 op/s 2024-04-16T05:52:02.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:02 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:02.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:02 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:02.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:02 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:52:02.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:02 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:52:02.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:02 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:02.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:02 smithi027 ceph-mon[93635]: pgmap v238: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 4.6 KiB/s rd, 4 op/s 2024-04-16T05:52:02.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:02 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:02.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:02 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:02.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:02 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:52:02.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:02 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:52:02.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:02 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:03.715 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:52:03.715 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.715 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.715 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.716 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.716 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.716 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: Upgrade: Setting container_image for all mds 2024-04-16T05:52:03.716 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:03.716 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi027.ejoytk"}]: dispatch 2024-04-16T05:52:03.716 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi027.ejoytk"}]': finished 2024-04-16T05:52:03.717 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi027.hkhpuq"}]: dispatch 2024-04-16T05:52:03.717 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi027.hkhpuq"}]': finished 2024-04-16T05:52:03.717 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi122.lojjdp"}]: dispatch 2024-04-16T05:52:03.717 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi122.lojjdp"}]': finished 2024-04-16T05:52:03.717 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi122.rulewi"}]: dispatch 2024-04-16T05:52:03.717 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi122.rulewi"}]': finished 2024-04-16T05:52:03.717 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.720 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: Upgrade: Setting container_image for all rgw 2024-04-16T05:52:03.720 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:03.720 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.721 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:03.721 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.721 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.721 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:03.721 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.721 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:03.721 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.721 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:03.721 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.721 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:03.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:52:03.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: Upgrade: Setting container_image for all mds 2024-04-16T05:52:03.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:03.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi027.ejoytk"}]: dispatch 2024-04-16T05:52:03.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi027.ejoytk"}]': finished 2024-04-16T05:52:03.774 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi027.hkhpuq"}]: dispatch 2024-04-16T05:52:03.775 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi027.hkhpuq"}]': finished 2024-04-16T05:52:03.775 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi122.lojjdp"}]: dispatch 2024-04-16T05:52:03.775 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi122.lojjdp"}]': finished 2024-04-16T05:52:03.775 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi122.rulewi"}]: dispatch 2024-04-16T05:52:03.775 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.cephfs.smithi122.rulewi"}]': finished 2024-04-16T05:52:03.775 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.775 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: Upgrade: Setting container_image for all rgw 2024-04-16T05:52:03.775 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:03.775 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.775 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:03.775 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.775 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.775 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:03.775 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.776 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:03.776 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.776 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:03.776 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:03.776 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:04.722 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:04 smithi027 ceph-mon[93635]: Upgrade: Setting container_image for all rbd-mirror 2024-04-16T05:52:04.722 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:04 smithi027 ceph-mon[93635]: Upgrade: Setting container_image for all ceph-exporter 2024-04-16T05:52:04.722 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:04 smithi027 ceph-mon[93635]: Upgrade: Setting container_image for all iscsi 2024-04-16T05:52:04.722 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:04 smithi027 ceph-mon[93635]: Upgrade: Setting container_image for all nfs 2024-04-16T05:52:04.722 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:04 smithi027 ceph-mon[93635]: Upgrade: Setting container_image for all nvmeof 2024-04-16T05:52:04.722 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:04 smithi027 ceph-mon[93635]: pgmap v239: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 1 op/s 2024-04-16T05:52:04.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:04 smithi122 ceph-mon[80930]: Upgrade: Setting container_image for all rbd-mirror 2024-04-16T05:52:04.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:04 smithi122 ceph-mon[80930]: Upgrade: Setting container_image for all ceph-exporter 2024-04-16T05:52:04.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:04 smithi122 ceph-mon[80930]: Upgrade: Setting container_image for all iscsi 2024-04-16T05:52:04.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:04 smithi122 ceph-mon[80930]: Upgrade: Setting container_image for all nfs 2024-04-16T05:52:04.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:04 smithi122 ceph-mon[80930]: Upgrade: Setting container_image for all nvmeof 2024-04-16T05:52:04.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:04 smithi122 ceph-mon[80930]: pgmap v239: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 1 op/s 2024-04-16T05:52:06.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:06 smithi122 ceph-mon[80930]: Upgrade: Updating node-exporter.smithi027 (1/2) 2024-04-16T05:52:06.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:06 smithi122 ceph-mon[80930]: Deploying daemon node-exporter.smithi027 on smithi027 2024-04-16T05:52:06.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:06 smithi122 ceph-mon[80930]: pgmap v240: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:06.773 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:06 smithi027 ceph-mon[93635]: Upgrade: Updating node-exporter.smithi027 (1/2) 2024-04-16T05:52:06.773 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:06 smithi027 ceph-mon[93635]: Deploying daemon node-exporter.smithi027 on smithi027 2024-04-16T05:52:06.774 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:06 smithi027 ceph-mon[93635]: pgmap v240: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:08.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.247+0000 7fa411600700 1 -- 172.21.15.27:0/2793167442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa40c0feff0 msgr2=0x7fa40c0ff400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:08.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.247+0000 7fa411600700 1 --2- 172.21.15.27:0/2793167442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa40c0feff0 0x7fa40c0ff400 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc004940 tx=0x7fa3fc009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:08.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.247+0000 7fa411600700 1 -- 172.21.15.27:0/2793167442 shutdown_connections 2024-04-16T05:52:08.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.247+0000 7fa411600700 1 --2- 172.21.15.27:0/2793167442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa40c1002a0 0x7fa40c100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:08.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.247+0000 7fa411600700 1 --2- 172.21.15.27:0/2793167442 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa40c0feff0 0x7fa40c0ff400 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:08.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.247+0000 7fa411600700 1 -- 172.21.15.27:0/2793167442 >> 172.21.15.27:0/2793167442 conn(0x7fa40c0fa790 msgr2=0x7fa40c0fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:08.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.247+0000 7fa411600700 1 -- 172.21.15.27:0/2793167442 shutdown_connections 2024-04-16T05:52:08.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.247+0000 7fa411600700 1 -- 172.21.15.27:0/2793167442 wait complete. 2024-04-16T05:52:08.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.248+0000 7fa411600700 1 Processor -- start 2024-04-16T05:52:08.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.248+0000 7fa411600700 1 -- start start 2024-04-16T05:52:08.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.249+0000 7fa411600700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa40c0feff0 0x7fa40c10fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:08.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.249+0000 7fa411600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa40c1002a0 0x7fa40c110050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:08.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.249+0000 7fa411600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40c110650 con 0x7fa40c1002a0 2024-04-16T05:52:08.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.249+0000 7fa411600700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40c1107c0 con 0x7fa40c0feff0 2024-04-16T05:52:08.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.249+0000 7fa40a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa40c1002a0 0x7fa40c110050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:08.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.249+0000 7fa40a000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa40c1002a0 0x7fa40c110050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49408/0 (socket says 172.21.15.27:49408) 2024-04-16T05:52:08.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.249+0000 7fa40a000700 1 -- 172.21.15.27:0/2955545298 learned_addr learned my addr 172.21.15.27:0/2955545298 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:52:08.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.250+0000 7fa40aa00700 1 --2- 172.21.15.27:0/2955545298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa40c0feff0 0x7fa40c10fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:08.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.250+0000 7fa40a000700 1 -- 172.21.15.27:0/2955545298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa40c0feff0 msgr2=0x7fa40c10fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:08.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.250+0000 7fa40a000700 1 --2- 172.21.15.27:0/2955545298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa40c0feff0 0x7fa40c10fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:08.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.250+0000 7fa40a000700 1 -- 172.21.15.27:0/2955545298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc005040 con 0x7fa40c1002a0 2024-04-16T05:52:08.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.250+0000 7fa40a000700 1 --2- 172.21.15.27:0/2955545298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa40c1002a0 0x7fa40c110050 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800c8c0 tx=0x7fa3f80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:08.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.250+0000 7fa402a00700 1 -- 172.21.15.27:0/2955545298 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3f800a980 con 0x7fa40c1002a0 2024-04-16T05:52:08.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.251+0000 7fa402a00700 1 -- 172.21.15.27:0/2955545298 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3f800ce60 con 0x7fa40c1002a0 2024-04-16T05:52:08.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.251+0000 7fa402a00700 1 -- 172.21.15.27:0/2955545298 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3f80115e0 con 0x7fa40c1002a0 2024-04-16T05:52:08.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.251+0000 7fa411600700 1 -- 172.21.15.27:0/2955545298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa40c115220 con 0x7fa40c1002a0 2024-04-16T05:52:08.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.251+0000 7fa411600700 1 -- 172.21.15.27:0/2955545298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa40c1157e0 con 0x7fa40c1002a0 2024-04-16T05:52:08.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.252+0000 7fa411600700 1 -- 172.21.15.27:0/2955545298 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa40c04c710 con 0x7fa40c1002a0 2024-04-16T05:52:08.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.253+0000 7fa402a00700 1 -- 172.21.15.27:0/2955545298 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa3f8012400 con 0x7fa40c1002a0 2024-04-16T05:52:08.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.255+0000 7fa402a00700 1 --2- 172.21.15.27:0/2955545298 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa3f006ec00 0x7fa3f00710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:08.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.255+0000 7fa402a00700 1 -- 172.21.15.27:0/2955545298 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7fa3f808eb30 con 0x7fa40c1002a0 2024-04-16T05:52:08.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.255+0000 7fa40aa00700 1 --2- 172.21.15.27:0/2955545298 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa3f006ec00 0x7fa3f00710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:08.254 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.256+0000 7fa40aa00700 1 --2- 172.21.15.27:0/2955545298 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa3f006ec00 0x7fa3f00710b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc004f90 tx=0x7fa3fc009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:08.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.259+0000 7fa402a00700 1 -- 172.21.15.27:0/2955545298 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa3f805b270 con 0x7fa40c1002a0 2024-04-16T05:52:08.540 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.539+0000 7fa411600700 1 -- 172.21.15.27:0/2955545298 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa40c104590 con 0x7fa3f006ec00 2024-04-16T05:52:08.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.542+0000 7fa402a00700 1 -- 172.21.15.27:0/2955545298 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+441 (secure 0 0 0) 0x7fa40c104590 con 0x7fa3f006ec00 2024-04-16T05:52:08.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.545+0000 7fa411600700 1 -- 172.21.15.27:0/2955545298 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa3f006ec00 msgr2=0x7fa3f00710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:08.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.545+0000 7fa411600700 1 --2- 172.21.15.27:0/2955545298 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa3f006ec00 0x7fa3f00710b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc004f90 tx=0x7fa3fc009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:08.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.545+0000 7fa411600700 1 -- 172.21.15.27:0/2955545298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa40c1002a0 msgr2=0x7fa40c110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:08.544 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.545+0000 7fa411600700 1 --2- 172.21.15.27:0/2955545298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa40c1002a0 0x7fa40c110050 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800c8c0 tx=0x7fa3f80044c0 comp rx=0 tx=0).stop 2024-04-16T05:52:08.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.545+0000 7fa411600700 1 -- 172.21.15.27:0/2955545298 shutdown_connections 2024-04-16T05:52:08.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.545+0000 7fa411600700 1 --2- 172.21.15.27:0/2955545298 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa3f006ec00 0x7fa3f00710b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:08.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.545+0000 7fa411600700 1 --2- 172.21.15.27:0/2955545298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa40c0feff0 0x7fa40c10fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:08.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.545+0000 7fa411600700 1 --2- 172.21.15.27:0/2955545298 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa40c1002a0 0x7fa40c110050 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:08.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.545+0000 7fa411600700 1 -- 172.21.15.27:0/2955545298 >> 172.21.15.27:0/2955545298 conn(0x7fa40c0fa790 msgr2=0x7fa40c0fbe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:08.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.546+0000 7fa411600700 1 -- 172.21.15.27:0/2955545298 shutdown_connections 2024-04-16T05:52:08.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.546+0000 7fa411600700 1 -- 172.21.15.27:0/2955545298 wait complete. 2024-04-16T05:52:08.560 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:52:08.715 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:08 smithi027 ceph-mon[93635]: pgmap v241: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:08.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.716+0000 7f5eca200700 1 -- 172.21.15.27:0/1321424073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec40fcdc0 msgr2=0x7f5ec40fd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:08.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.716+0000 7f5eca200700 1 --2- 172.21.15.27:0/1321424073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec40fcdc0 0x7f5ec40fd1d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4004940 tx=0x7f5eb4009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:08.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.716+0000 7f5eca200700 1 -- 172.21.15.27:0/1321424073 shutdown_connections 2024-04-16T05:52:08.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.716+0000 7f5eca200700 1 --2- 172.21.15.27:0/1321424073 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ec40fd710 0x7f5ec40fe070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:08.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.716+0000 7f5eca200700 1 --2- 172.21.15.27:0/1321424073 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec40fcdc0 0x7f5ec40fd1d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:08.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.716+0000 7f5eca200700 1 -- 172.21.15.27:0/1321424073 >> 172.21.15.27:0/1321424073 conn(0x7f5ec40f8540 msgr2=0x7f5ec40fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:08.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.716+0000 7f5eca200700 1 -- 172.21.15.27:0/1321424073 shutdown_connections 2024-04-16T05:52:08.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.716+0000 7f5eca200700 1 -- 172.21.15.27:0/1321424073 wait complete. 2024-04-16T05:52:08.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.718+0000 7f5eca200700 1 Processor -- start 2024-04-16T05:52:08.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.718+0000 7f5eca200700 1 -- start start 2024-04-16T05:52:08.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.718+0000 7f5eca200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ec40fcdc0 0x7f5ec410b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:08.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.718+0000 7f5eca200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec40fd710 0x7f5ec410bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:08.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.718+0000 7f5eca200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec410c3d0 con 0x7f5ec40fd710 2024-04-16T05:52:08.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.718+0000 7f5eca200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec410cc20 con 0x7f5ec40fcdc0 2024-04-16T05:52:08.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.719+0000 7f5ec2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec40fd710 0x7f5ec410bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:08.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.719+0000 7f5ec2a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec40fd710 0x7f5ec410bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49426/0 (socket says 172.21.15.27:49426) 2024-04-16T05:52:08.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.719+0000 7f5ec2a00700 1 -- 172.21.15.27:0/4212733756 learned_addr learned my addr 172.21.15.27:0/4212733756 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:52:08.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.719+0000 7f5ec3400700 1 --2- 172.21.15.27:0/4212733756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ec40fcdc0 0x7f5ec410b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:08.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.719+0000 7f5ec2a00700 1 -- 172.21.15.27:0/4212733756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ec40fcdc0 msgr2=0x7f5ec410b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:08.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.719+0000 7f5ec2a00700 1 --2- 172.21.15.27:0/4212733756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ec40fcdc0 0x7f5ec410b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:08.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.719+0000 7f5ec2a00700 1 -- 172.21.15.27:0/4212733756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb4005040 con 0x7f5ec40fd710 2024-04-16T05:52:08.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.720+0000 7f5ec2a00700 1 --2- 172.21.15.27:0/4212733756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec40fd710 0x7f5ec410bd80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5eb000c800 tx=0x7f5eb0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:08.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.720+0000 7f5ebb400700 1 -- 172.21.15.27:0/4212733756 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5eb000a760 con 0x7f5ec40fd710 2024-04-16T05:52:08.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.720+0000 7f5ebb400700 1 -- 172.21.15.27:0/4212733756 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5eb000ad90 con 0x7f5ec40fd710 2024-04-16T05:52:08.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.720+0000 7f5eca200700 1 -- 172.21.15.27:0/4212733756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec410cf00 con 0x7f5ec40fd710 2024-04-16T05:52:08.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.720+0000 7f5eca200700 1 -- 172.21.15.27:0/4212733756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec410d490 con 0x7f5ec40fd710 2024-04-16T05:52:08.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.721+0000 7f5ebb400700 1 -- 172.21.15.27:0/4212733756 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5eb00113a0 con 0x7f5ec40fd710 2024-04-16T05:52:08.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.722+0000 7f5eca200700 1 -- 172.21.15.27:0/4212733756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea40052e0 con 0x7f5ec40fd710 2024-04-16T05:52:08.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.723+0000 7f5ebb400700 1 -- 172.21.15.27:0/4212733756 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5eb000a8c0 con 0x7f5ec40fd710 2024-04-16T05:52:08.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.724+0000 7f5ebb400700 1 --2- 172.21.15.27:0/4212733756 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5ea806ec10 0x7f5ea80710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:08.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.725+0000 7f5ebb400700 1 -- 172.21.15.27:0/4212733756 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f5eb008eab0 con 0x7f5ec40fd710 2024-04-16T05:52:08.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.725+0000 7f5ec3400700 1 --2- 172.21.15.27:0/4212733756 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5ea806ec10 0x7f5ea80710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:08.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.726+0000 7f5ec3400700 1 --2- 172.21.15.27:0/4212733756 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5ea806ec10 0x7f5ea80710c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4000c00 tx=0x7f5eb4009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.729+0000 7f5ebb400700 1 -- 172.21.15.27:0/4212733756 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5eb0057680 con 0x7f5ec40fd710 2024-04-16T05:52:08.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:08 smithi122 ceph-mon[80930]: pgmap v241: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:08.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.994+0000 7f5eca200700 1 -- 172.21.15.27:0/4212733756 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ea4000bf0 con 0x7f5ea806ec10 2024-04-16T05:52:08.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:08.997+0000 7f5ebb400700 1 -- 172.21.15.27:0/4212733756 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+441 (secure 0 0 0) 0x7f5ea4000bf0 con 0x7f5ea806ec10 2024-04-16T05:52:08.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.000+0000 7f5eca200700 1 -- 172.21.15.27:0/4212733756 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5ea806ec10 msgr2=0x7f5ea80710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:08.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.000+0000 7f5eca200700 1 --2- 172.21.15.27:0/4212733756 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5ea806ec10 0x7f5ea80710c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5eb4000c00 tx=0x7f5eb4009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:08.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.000+0000 7f5eca200700 1 -- 172.21.15.27:0/4212733756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec40fd710 msgr2=0x7f5ec410bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:08.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.000+0000 7f5eca200700 1 --2- 172.21.15.27:0/4212733756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec40fd710 0x7f5ec410bd80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5eb000c800 tx=0x7f5eb0004470 comp rx=0 tx=0).stop 2024-04-16T05:52:08.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.000+0000 7f5eca200700 1 -- 172.21.15.27:0/4212733756 shutdown_connections 2024-04-16T05:52:09.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.000+0000 7f5eca200700 1 --2- 172.21.15.27:0/4212733756 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5ea806ec10 0x7f5ea80710c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:09.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.000+0000 7f5eca200700 1 --2- 172.21.15.27:0/4212733756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ec40fcdc0 0x7f5ec410b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:09.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.000+0000 7f5eca200700 1 --2- 172.21.15.27:0/4212733756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ec40fd710 0x7f5ec410bd80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:09.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.000+0000 7f5eca200700 1 -- 172.21.15.27:0/4212733756 >> 172.21.15.27:0/4212733756 conn(0x7f5ec40f8540 msgr2=0x7f5ec40f9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:09.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.000+0000 7f5eca200700 1 -- 172.21.15.27:0/4212733756 shutdown_connections 2024-04-16T05:52:09.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.001+0000 7f5eca200700 1 -- 172.21.15.27:0/4212733756 wait complete. 2024-04-16T05:52:09.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.157+0000 7f2115a00700 1 -- 172.21.15.27:0/1436929814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2110070890 msgr2=0x7f2110070d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:09.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.157+0000 7f2115a00700 1 --2- 172.21.15.27:0/1436929814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2110070890 0x7f2110070d00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f20fc004940 tx=0x7f20fc009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:09.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.157+0000 7f2115a00700 1 -- 172.21.15.27:0/1436929814 shutdown_connections 2024-04-16T05:52:09.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.157+0000 7f2115a00700 1 --2- 172.21.15.27:0/1436929814 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2110070890 0x7f2110070d00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:09.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.157+0000 7f2115a00700 1 --2- 172.21.15.27:0/1436929814 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2110071e50 0x7f2110070350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:09.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.157+0000 7f2115a00700 1 -- 172.21.15.27:0/1436929814 >> 172.21.15.27:0/1436929814 conn(0x7f21100f8540 msgr2=0x7f21100fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:09.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.157+0000 7f2115a00700 1 -- 172.21.15.27:0/1436929814 shutdown_connections 2024-04-16T05:52:09.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.157+0000 7f2115a00700 1 -- 172.21.15.27:0/1436929814 wait complete. 2024-04-16T05:52:09.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.158+0000 7f2115a00700 1 Processor -- start 2024-04-16T05:52:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.158+0000 7f2115a00700 1 -- start start 2024-04-16T05:52:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.159+0000 7f2115a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2110070890 0x7f211010fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.159+0000 7f2115a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2110071e50 0x7f2110110190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.159+0000 7f2115a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2110110790 con 0x7f2110071e50 2024-04-16T05:52:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.159+0000 7f2115a00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2110110fe0 con 0x7f2110070890 2024-04-16T05:52:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.159+0000 7f210e000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2110071e50 0x7f2110110190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.159+0000 7f210e000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2110071e50 0x7f2110110190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49438/0 (socket says 172.21.15.27:49438) 2024-04-16T05:52:09.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.159+0000 7f210e000700 1 -- 172.21.15.27:0/1075126225 learned_addr learned my addr 172.21.15.27:0/1075126225 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:52:09.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.159+0000 7f210ea00700 1 --2- 172.21.15.27:0/1075126225 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2110070890 0x7f211010fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:09.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.159+0000 7f210e000700 1 -- 172.21.15.27:0/1075126225 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2110070890 msgr2=0x7f211010fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:09.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.159+0000 7f210e000700 1 --2- 172.21.15.27:0/1075126225 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2110070890 0x7f211010fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:09.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.159+0000 7f210e000700 1 -- 172.21.15.27:0/1075126225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20fc005040 con 0x7f2110071e50 2024-04-16T05:52:09.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.160+0000 7f210e000700 1 --2- 172.21.15.27:0/1075126225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2110071e50 0x7f2110110190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00b3b0 tx=0x7f20fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:09.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.160+0000 7f2106a00700 1 -- 172.21.15.27:0/1075126225 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20fc00e040 con 0x7f2110071e50 2024-04-16T05:52:09.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.160+0000 7f2106a00700 1 -- 172.21.15.27:0/1075126225 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f20fc00bcc0 con 0x7f2110071e50 2024-04-16T05:52:09.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.160+0000 7f2106a00700 1 -- 172.21.15.27:0/1075126225 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20fc01c630 con 0x7f2110071e50 2024-04-16T05:52:09.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.160+0000 7f2115a00700 1 -- 172.21.15.27:0/1075126225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2110111260 con 0x7f2110071e50 2024-04-16T05:52:09.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.160+0000 7f2115a00700 1 -- 172.21.15.27:0/1075126225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21101117a0 con 0x7f2110071e50 2024-04-16T05:52:09.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.161+0000 7f2115a00700 1 -- 172.21.15.27:0/1075126225 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f211005f5f0 con 0x7f2110071e50 2024-04-16T05:52:09.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.162+0000 7f2106a00700 1 -- 172.21.15.27:0/1075126225 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f20fc013980 con 0x7f2110071e50 2024-04-16T05:52:09.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.164+0000 7f2106a00700 1 --2- 172.21.15.27:0/1075126225 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f20f406ec00 0x7f20f40710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:09.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.164+0000 7f2106a00700 1 -- 172.21.15.27:0/1075126225 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f20fc08f790 con 0x7f2110071e50 2024-04-16T05:52:09.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.164+0000 7f210ea00700 1 --2- 172.21.15.27:0/1075126225 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f20f406ec00 0x7f20f40710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:09.163 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.165+0000 7f210ea00700 1 --2- 172.21.15.27:0/1075126225 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f20f406ec00 0x7f20f40710b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2100007fd0 tx=0x7f2100009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:09.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.169+0000 7f2106a00700 1 -- 172.21.15.27:0/1075126225 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f20fc05be50 con 0x7f2110071e50 2024-04-16T05:52:09.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.425+0000 7f2115a00700 1 -- 172.21.15.27:0/1075126225 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2110111be0 con 0x7f20f406ec00 2024-04-16T05:52:09.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.443+0000 7f2106a00700 1 -- 172.21.15.27:0/1075126225 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f2110111be0 con 0x7f20f406ec00 2024-04-16T05:52:09.442 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:52:09.443 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (8m) 67s ago 19m 13.5M - 0.25.0 c8568f914cd2 4ed63a8a6b33 2024-04-16T05:52:09.443 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (5m) 67s ago 19m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 4aa1f0006685 2024-04-16T05:52:09.443 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (5m) 14s ago 18m 6656k - 19.0.0-2884-g218a6914 a4e4811815e4 1c1912507d56 2024-04-16T05:52:09.443 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (8m) 67s ago 18m 74.0M - 9.4.7 954c08fa6188 f7ee2a1afd12 2024-04-16T05:52:09.443 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (74s) 67s ago 11m 11.0M - 19.0.0-2884-g218a6914 a4e4811815e4 b3c2178c537e 2024-04-16T05:52:09.443 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (105s) 67s ago 11m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 14b0c83f2891 2024-04-16T05:52:09.443 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (43s) 14s ago 11m 11.0M - 19.0.0-2884-g218a6914 a4e4811815e4 8ba95adfaaef 2024-04-16T05:52:09.443 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (18s) 14s ago 11m 10.9M - 19.0.0-2884-g218a6914 a4e4811815e4 c60e9ca0f1c8 2024-04-16T05:52:09.443 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:8443,9283,8765 running (8m) 67s ago 20m 509M - 19.0.0-2884-g218a6914 a4e4811815e4 b69a0b08e12b 2024-04-16T05:52:09.443 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283,8765 running (7m) 14s ago 17m 454M - 19.0.0-2884-g218a6914 a4e4811815e4 32e27bcd7e42 2024-04-16T05:52:09.443 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (6m) 67s ago 21m 53.4M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 299ca337e4ca 2024-04-16T05:52:09.443 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (6m) 14s ago 17m 40.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5db5f3037ed0 2024-04-16T05:52:09.443 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (8m) 67s ago 19m 7994k - 1.5.0 0da6a335fe13 5e29e29868b0 2024-04-16T05:52:09.444 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (8m) 14s ago 17m 8284k - 1.5.0 0da6a335fe13 a360922bc621 2024-04-16T05:52:09.444 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (4m) 67s ago 16m 45.9M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 cef03b67bc5a 2024-04-16T05:52:09.444 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (4m) 67s ago 16m 44.8M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3a6ab1fc6e79 2024-04-16T05:52:09.444 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (3m) 67s ago 15m 43.3M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3524731d7277 2024-04-16T05:52:09.444 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (3m) 14s ago 14m 44.6M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 897d3579f929 2024-04-16T05:52:09.444 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (2m) 14s ago 14m 42.1M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 fa97758ff7c5 2024-04-16T05:52:09.444 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (2m) 14s ago 13m 49.5M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 fd12260a820d 2024-04-16T05:52:09.444 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (7m) 67s ago 18m 59.2M - 2.43.0 a07b618ecd1d e70f55ba5ec0 2024-04-16T05:52:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.445+0000 7f2115a00700 1 -- 172.21.15.27:0/1075126225 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f20f406ec00 msgr2=0x7f20f40710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.445+0000 7f2115a00700 1 --2- 172.21.15.27:0/1075126225 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f20f406ec00 0x7f20f40710b0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2100007fd0 tx=0x7f2100009040 comp rx=0 tx=0).stop 2024-04-16T05:52:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.445+0000 7f2115a00700 1 -- 172.21.15.27:0/1075126225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2110071e50 msgr2=0x7f2110110190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.445+0000 7f2115a00700 1 --2- 172.21.15.27:0/1075126225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2110071e50 0x7f2110110190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00b3b0 tx=0x7f20fc009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.446+0000 7f2115a00700 1 -- 172.21.15.27:0/1075126225 shutdown_connections 2024-04-16T05:52:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.446+0000 7f2115a00700 1 --2- 172.21.15.27:0/1075126225 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f20f406ec00 0x7f20f40710b0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.446+0000 7f2115a00700 1 --2- 172.21.15.27:0/1075126225 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2110070890 0x7f211010fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.446+0000 7f2115a00700 1 --2- 172.21.15.27:0/1075126225 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2110071e50 0x7f2110110190 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:09.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.446+0000 7f2115a00700 1 -- 172.21.15.27:0/1075126225 >> 172.21.15.27:0/1075126225 conn(0x7f21100f8540 msgr2=0x7f21100f9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.446+0000 7f2115a00700 1 -- 172.21.15.27:0/1075126225 shutdown_connections 2024-04-16T05:52:09.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.446+0000 7f2115a00700 1 -- 172.21.15.27:0/1075126225 wait complete. 2024-04-16T05:52:09.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.604+0000 7f253ac00700 1 -- 172.21.15.27:0/3338589189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25340ff020 msgr2=0x7f25340ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:09.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.604+0000 7f253ac00700 1 --2- 172.21.15.27:0/3338589189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25340ff020 0x7f25340ff430 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2524004940 tx=0x7f2524009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:09.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.605+0000 7f253ac00700 1 -- 172.21.15.27:0/3338589189 shutdown_connections 2024-04-16T05:52:09.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.605+0000 7f253ac00700 1 --2- 172.21.15.27:0/3338589189 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25341002d0 0x7f2534100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:09.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.605+0000 7f253ac00700 1 --2- 172.21.15.27:0/3338589189 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25340ff020 0x7f25340ff430 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:09.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.605+0000 7f253ac00700 1 -- 172.21.15.27:0/3338589189 >> 172.21.15.27:0/3338589189 conn(0x7f25340fa950 msgr2=0x7f25340fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:09.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.605+0000 7f253ac00700 1 -- 172.21.15.27:0/3338589189 shutdown_connections 2024-04-16T05:52:09.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.605+0000 7f253ac00700 1 -- 172.21.15.27:0/3338589189 wait complete. 2024-04-16T05:52:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.606+0000 7f253ac00700 1 Processor -- start 2024-04-16T05:52:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.606+0000 7f253ac00700 1 -- start start 2024-04-16T05:52:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.606+0000 7f253ac00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25340ff020 0x7f253410fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:09.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.606+0000 7f253ac00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25341002d0 0x7f25341100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:09.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.606+0000 7f253ac00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25341106a0 con 0x7f25341002d0 2024-04-16T05:52:09.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.606+0000 7f253ac00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2534110810 con 0x7f25340ff020 2024-04-16T05:52:09.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.607+0000 7f2533400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25341002d0 0x7f25341100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:09.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.607+0000 7f2533400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25341002d0 0x7f25341100a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49444/0 (socket says 172.21.15.27:49444) 2024-04-16T05:52:09.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.607+0000 7f2533400700 1 -- 172.21.15.27:0/3386252529 learned_addr learned my addr 172.21.15.27:0/3386252529 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:52:09.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.607+0000 7f2533400700 1 -- 172.21.15.27:0/3386252529 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25340ff020 msgr2=0x7f253410fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:09.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.607+0000 7f2533400700 1 --2- 172.21.15.27:0/3386252529 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25340ff020 0x7f253410fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:09.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.607+0000 7f2533400700 1 -- 172.21.15.27:0/3386252529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2524005040 con 0x7f25341002d0 2024-04-16T05:52:09.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.607+0000 7f2533400700 1 --2- 172.21.15.27:0/3386252529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25341002d0 0x7f25341100a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f252000c8c0 tx=0x7f25200044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:09.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.608+0000 7f252be00700 1 -- 172.21.15.27:0/3386252529 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f252000a980 con 0x7f25341002d0 2024-04-16T05:52:09.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.608+0000 7f252be00700 1 -- 172.21.15.27:0/3386252529 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f252000ce60 con 0x7f25341002d0 2024-04-16T05:52:09.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.608+0000 7f252be00700 1 -- 172.21.15.27:0/3386252529 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25200115e0 con 0x7f25341002d0 2024-04-16T05:52:09.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.608+0000 7f253ac00700 1 -- 172.21.15.27:0/3386252529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2534115270 con 0x7f25341002d0 2024-04-16T05:52:09.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.608+0000 7f253ac00700 1 -- 172.21.15.27:0/3386252529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2534115830 con 0x7f25341002d0 2024-04-16T05:52:09.608 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.609+0000 7f253ac00700 1 -- 172.21.15.27:0/3386252529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f253404c710 con 0x7f25341002d0 2024-04-16T05:52:09.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.611+0000 7f252be00700 1 -- 172.21.15.27:0/3386252529 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2520012400 con 0x7f25341002d0 2024-04-16T05:52:09.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.612+0000 7f252be00700 1 --2- 172.21.15.27:0/3386252529 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f251806ec00 0x7f25180710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:09.611 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.612+0000 7f252be00700 1 -- 172.21.15.27:0/3386252529 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f252008eb30 con 0x7f25341002d0 2024-04-16T05:52:09.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.613+0000 7f2533e00700 1 --2- 172.21.15.27:0/3386252529 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f251806ec00 0x7f25180710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:09.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.614+0000 7f2533e00700 1 --2- 172.21.15.27:0/3386252529 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f251806ec00 0x7f25180710b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2524004f90 tx=0x7f2524009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:09.619 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.621+0000 7f252be00700 1 -- 172.21.15.27:0/3386252529 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f252005b270 con 0x7f25341002d0 2024-04-16T05:52:09.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:09 smithi122 ceph-mon[80930]: from='client.34496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:52:09.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:09 smithi027 ceph-mon[93635]: from='client.34496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:52:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.939+0000 7f253ac00700 1 -- 172.21.15.27:0/3386252529 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f2534115ce0 con 0x7f25341002d0 2024-04-16T05:52:09.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.940+0000 7f252be00700 1 -- 172.21.15.27:0/3386252529 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f25200126b0 con 0x7f25341002d0 2024-04-16T05:52:09.939 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:52:09.939 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:52:09.939 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:52:09.939 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:52:09.939 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:52:09.939 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:52:09.939 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:52:09.939 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:52:09.939 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T05:52:09.939 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:52:09.940 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:52:09.940 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T05:52:09.940 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:52:09.940 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:52:09.940 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 14 2024-04-16T05:52:09.940 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:52:09.940 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:52:09.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.942+0000 7f253ac00700 1 -- 172.21.15.27:0/3386252529 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f251806ec00 msgr2=0x7f25180710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:09.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.942+0000 7f253ac00700 1 --2- 172.21.15.27:0/3386252529 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f251806ec00 0x7f25180710b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2524004f90 tx=0x7f2524009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:09.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.942+0000 7f253ac00700 1 -- 172.21.15.27:0/3386252529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25341002d0 msgr2=0x7f25341100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:09.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.942+0000 7f253ac00700 1 --2- 172.21.15.27:0/3386252529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25341002d0 0x7f25341100a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f252000c8c0 tx=0x7f25200044c0 comp rx=0 tx=0).stop 2024-04-16T05:52:09.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.943+0000 7f253ac00700 1 -- 172.21.15.27:0/3386252529 shutdown_connections 2024-04-16T05:52:09.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.943+0000 7f253ac00700 1 --2- 172.21.15.27:0/3386252529 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f251806ec00 0x7f25180710b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.943+0000 7f253ac00700 1 --2- 172.21.15.27:0/3386252529 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f25340ff020 0x7f253410fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.943+0000 7f253ac00700 1 --2- 172.21.15.27:0/3386252529 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f25341002d0 0x7f25341100a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.943+0000 7f253ac00700 1 -- 172.21.15.27:0/3386252529 >> 172.21.15.27:0/3386252529 conn(0x7f25340fa950 msgr2=0x7f25340fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.943+0000 7f253ac00700 1 -- 172.21.15.27:0/3386252529 shutdown_connections 2024-04-16T05:52:09.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:09.943+0000 7f253ac00700 1 -- 172.21.15.27:0/3386252529 wait complete. 2024-04-16T05:52:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.101+0000 7fd512000700 1 -- 172.21.15.27:0/2286287730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd50c0fdee0 msgr2=0x7fd50c0fe350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:10.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.101+0000 7fd512000700 1 --2- 172.21.15.27:0/2286287730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd50c0fdee0 0x7fd50c0fe350 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc004940 tx=0x7fd4fc009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.101+0000 7fd512000700 1 -- 172.21.15.27:0/2286287730 shutdown_connections 2024-04-16T05:52:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.101+0000 7fd512000700 1 --2- 172.21.15.27:0/2286287730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd50c0fdee0 0x7fd50c0fe350 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.101+0000 7fd512000700 1 --2- 172.21.15.27:0/2286287730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd50c0fcc30 0x7fd50c0fd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.101+0000 7fd512000700 1 -- 172.21.15.27:0/2286287730 >> 172.21.15.27:0/2286287730 conn(0x7fd50c0f83d0 msgr2=0x7fd50c0fa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.101+0000 7fd512000700 1 -- 172.21.15.27:0/2286287730 shutdown_connections 2024-04-16T05:52:10.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.102+0000 7fd512000700 1 -- 172.21.15.27:0/2286287730 wait complete. 2024-04-16T05:52:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.102+0000 7fd512000700 1 Processor -- start 2024-04-16T05:52:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.103+0000 7fd512000700 1 -- start start 2024-04-16T05:52:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.103+0000 7fd512000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd50c0fcc30 0x7fd50c10fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.103+0000 7fd512000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd50c0fdee0 0x7fd50c10ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.103+0000 7fd512000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c1105a0 con 0x7fd50c0fdee0 2024-04-16T05:52:10.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.103+0000 7fd512000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c110df0 con 0x7fd50c0fcc30 2024-04-16T05:52:10.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.103+0000 7fd50be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd50c0fdee0 0x7fd50c10ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:10.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.103+0000 7fd50be00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd50c0fdee0 0x7fd50c10ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49452/0 (socket says 172.21.15.27:49452) 2024-04-16T05:52:10.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.103+0000 7fd50be00700 1 -- 172.21.15.27:0/743008933 learned_addr learned my addr 172.21.15.27:0/743008933 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:52:10.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.104+0000 7fd510c00700 1 --2- 172.21.15.27:0/743008933 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd50c0fcc30 0x7fd50c10fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:10.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.104+0000 7fd50be00700 1 -- 172.21.15.27:0/743008933 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd50c0fcc30 msgr2=0x7fd50c10fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:10.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.104+0000 7fd50be00700 1 --2- 172.21.15.27:0/743008933 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd50c0fcc30 0x7fd50c10fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:10.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.104+0000 7fd50be00700 1 -- 172.21.15.27:0/743008933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4fc005040 con 0x7fd50c0fdee0 2024-04-16T05:52:10.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.104+0000 7fd510c00700 1 --2- 172.21.15.27:0/743008933 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd50c0fcc30 0x7fd50c10fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:52:10.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.104+0000 7fd50be00700 1 --2- 172.21.15.27:0/743008933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd50c0fdee0 0x7fd50c10ffa0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc00b3b0 tx=0x7fd4fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:10.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.105+0000 7fd508c00700 1 -- 172.21.15.27:0/743008933 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4fc00e040 con 0x7fd50c0fdee0 2024-04-16T05:52:10.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.105+0000 7fd512000700 1 -- 172.21.15.27:0/743008933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd50c111070 con 0x7fd50c0fdee0 2024-04-16T05:52:10.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.105+0000 7fd508c00700 1 -- 172.21.15.27:0/743008933 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd4fc004290 con 0x7fd50c0fdee0 2024-04-16T05:52:10.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.105+0000 7fd508c00700 1 -- 172.21.15.27:0/743008933 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4fc01c630 con 0x7fd50c0fdee0 2024-04-16T05:52:10.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.105+0000 7fd512000700 1 -- 172.21.15.27:0/743008933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd50c1115b0 con 0x7fd50c0fdee0 2024-04-16T05:52:10.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.106+0000 7fd512000700 1 -- 172.21.15.27:0/743008933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd50c101870 con 0x7fd50c0fdee0 2024-04-16T05:52:10.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.107+0000 7fd508c00700 1 -- 172.21.15.27:0/743008933 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd4fc00bbf0 con 0x7fd50c0fdee0 2024-04-16T05:52:10.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.108+0000 7fd508c00700 1 --2- 172.21.15.27:0/743008933 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd4f406ec10 0x7fd4f40710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:10.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.108+0000 7fd508c00700 1 -- 172.21.15.27:0/743008933 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7fd4fc08ff40 con 0x7fd50c0fdee0 2024-04-16T05:52:10.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.109+0000 7fd510c00700 1 --2- 172.21.15.27:0/743008933 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd4f406ec10 0x7fd4f40710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:10.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.109+0000 7fd510c00700 1 --2- 172.21.15.27:0/743008933 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd4f406ec10 0x7fd4f40710c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd500007fd0 tx=0x7fd500009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:10.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.113+0000 7fd508c00700 1 -- 172.21.15.27:0/743008933 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd4fc05c600 con 0x7fd50c0fdee0 2024-04-16T05:52:10.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.401+0000 7fd512000700 1 -- 172.21.15.27:0/743008933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd50c105870 con 0x7fd50c0fdee0 2024-04-16T05:52:10.401 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.402+0000 7fd508c00700 1 -- 172.21.15.27:0/743008933 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1936 (secure 0 0 0) 0x7fd4fc05c190 con 0x7fd50c0fdee0 2024-04-16T05:52:10.401 INFO:teuthology.orchestra.run.smithi027.stdout:e36 2024-04-16T05:52:10.402 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:52:10.402 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:52:10.402 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:52:10.402 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:52:10.402 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:52:10.402 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:52:10.402 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 35 2024-04-16T05:52:10.402 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:52:10.402 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:52:10.402 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:51:47.041625+0000 2024-04-16T05:52:10.402 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:52:10.402 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:52:10.402 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:52:10.403 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:52:10.403 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:52:10.403 INFO:teuthology.orchestra.run.smithi027.stdout:max_xattr_size 65536 2024-04-16T05:52:10.403 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:52:10.403 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:52:10.403 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 88 2024-04-16T05:52:10.403 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-16T05:52:10.403 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:52:10.403 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:52:10.403 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=34400} 2024-04-16T05:52:10.403 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:52:10.403 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:52:10.403 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:52:10.403 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:52:10.404 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:52:10.404 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:52:10.404 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:52:10.404 INFO:teuthology.orchestra.run.smithi027.stdout:bal_rank_mask -1 2024-04-16T05:52:10.404 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:52:10.404 INFO:teuthology.orchestra.run.smithi027.stdout:qdb_cluster leader: 34400 members: 34400 2024-04-16T05:52:10.404 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{0:34400} state up:active seq 23 join_fscid=1 addr [v2:172.21.15.27:6826/2831468256,v1:172.21.15.27:6827/2831468256] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:52:10.404 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:52:10.404 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:52:10.404 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:52:10.404 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:52:10.404 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{-1:34432} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6828/3219609959,v1:172.21.15.27:6829/3219609959] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:52:10.404 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:44337} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/97103507,v1:172.21.15.122:6825/97103507] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:52:10.404 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:44359} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/1030624816,v1:172.21.15.122:6827/1030624816] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:52:10.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.405+0000 7fd512000700 1 -- 172.21.15.27:0/743008933 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd4f406ec10 msgr2=0x7fd4f40710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:10.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.405+0000 7fd512000700 1 --2- 172.21.15.27:0/743008933 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd4f406ec10 0x7fd4f40710c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd500007fd0 tx=0x7fd500009040 comp rx=0 tx=0).stop 2024-04-16T05:52:10.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.405+0000 7fd512000700 1 -- 172.21.15.27:0/743008933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd50c0fdee0 msgr2=0x7fd50c10ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:10.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.405+0000 7fd512000700 1 --2- 172.21.15.27:0/743008933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd50c0fdee0 0x7fd50c10ffa0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc00b3b0 tx=0x7fd4fc009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:10.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.405+0000 7fd512000700 1 -- 172.21.15.27:0/743008933 shutdown_connections 2024-04-16T05:52:10.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.405+0000 7fd512000700 1 --2- 172.21.15.27:0/743008933 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd4f406ec10 0x7fd4f40710c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:10.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.405+0000 7fd512000700 1 --2- 172.21.15.27:0/743008933 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd50c0fcc30 0x7fd50c10fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:10.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.405+0000 7fd512000700 1 --2- 172.21.15.27:0/743008933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd50c0fdee0 0x7fd50c10ffa0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:10.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.405+0000 7fd512000700 1 -- 172.21.15.27:0/743008933 >> 172.21.15.27:0/743008933 conn(0x7fd50c0f83d0 msgr2=0x7fd50c0f9970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:10.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.406+0000 7fd512000700 1 -- 172.21.15.27:0/743008933 shutdown_connections 2024-04-16T05:52:10.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.406+0000 7fd512000700 1 -- 172.21.15.27:0/743008933 wait complete. 2024-04-16T05:52:10.406 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 36 2024-04-16T05:52:10.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.571+0000 7f7b86400700 1 -- 172.21.15.27:0/1032492027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b800708f0 msgr2=0x7f7b80070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:10.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.571+0000 7f7b86400700 1 --2- 172.21.15.27:0/1032492027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b800708f0 0x7f7b80070d60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c004940 tx=0x7f7b6c009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:10.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.571+0000 7f7b86400700 1 -- 172.21.15.27:0/1032492027 shutdown_connections 2024-04-16T05:52:10.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.571+0000 7f7b86400700 1 --2- 172.21.15.27:0/1032492027 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b800708f0 0x7f7b80070d60 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:10.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.571+0000 7f7b86400700 1 --2- 172.21.15.27:0/1032492027 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b80071ea0 0x7f7b800703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:10.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.571+0000 7f7b86400700 1 -- 172.21.15.27:0/1032492027 >> 172.21.15.27:0/1032492027 conn(0x7f7b800f8580 msgr2=0x7f7b800fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:10.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.572+0000 7f7b86400700 1 -- 172.21.15.27:0/1032492027 shutdown_connections 2024-04-16T05:52:10.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.572+0000 7f7b86400700 1 -- 172.21.15.27:0/1032492027 wait complete. 2024-04-16T05:52:10.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.572+0000 7f7b86400700 1 Processor -- start 2024-04-16T05:52:10.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.573+0000 7f7b86400700 1 -- start start 2024-04-16T05:52:10.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.573+0000 7f7b86400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b800708f0 0x7f7b80113fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:10.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.573+0000 7f7b86400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b80071ea0 0x7f7b80114520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:10.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.573+0000 7f7b86400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b80114b20 con 0x7f7b80071ea0 2024-04-16T05:52:10.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.573+0000 7f7b86400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b80114c90 con 0x7f7b800708f0 2024-04-16T05:52:10.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.573+0000 7f7b7ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b80071ea0 0x7f7b80114520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:10.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.573+0000 7f7b7ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b80071ea0 0x7f7b80114520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49464/0 (socket says 172.21.15.27:49464) 2024-04-16T05:52:10.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.573+0000 7f7b7ea00700 1 -- 172.21.15.27:0/4133240408 learned_addr learned my addr 172.21.15.27:0/4133240408 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:52:10.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.574+0000 7f7b7f400700 1 --2- 172.21.15.27:0/4133240408 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b800708f0 0x7f7b80113fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:10.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.574+0000 7f7b7ea00700 1 -- 172.21.15.27:0/4133240408 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b800708f0 msgr2=0x7f7b80113fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:10.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.574+0000 7f7b7ea00700 1 --2- 172.21.15.27:0/4133240408 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b800708f0 0x7f7b80113fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:10.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.574+0000 7f7b7ea00700 1 -- 172.21.15.27:0/4133240408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6c005040 con 0x7f7b80071ea0 2024-04-16T05:52:10.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.574+0000 7f7b7ea00700 1 --2- 172.21.15.27:0/4133240408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b80071ea0 0x7f7b80114520 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00b970 tx=0x7f7b6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:10.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.574+0000 7f7b77400700 1 -- 172.21.15.27:0/4133240408 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b6c00e040 con 0x7f7b80071ea0 2024-04-16T05:52:10.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.575+0000 7f7b77400700 1 -- 172.21.15.27:0/4133240408 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b6c004290 con 0x7f7b80071ea0 2024-04-16T05:52:10.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.575+0000 7f7b77400700 1 -- 172.21.15.27:0/4133240408 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b6c01c630 con 0x7f7b80071ea0 2024-04-16T05:52:10.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.575+0000 7f7b86400700 1 -- 172.21.15.27:0/4133240408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b80119640 con 0x7f7b80071ea0 2024-04-16T05:52:10.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.575+0000 7f7b86400700 1 -- 172.21.15.27:0/4133240408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b80119c00 con 0x7f7b80071ea0 2024-04-16T05:52:10.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.576+0000 7f7b86400700 1 -- 172.21.15.27:0/4133240408 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b8004c710 con 0x7f7b80071ea0 2024-04-16T05:52:10.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.577+0000 7f7b77400700 1 -- 172.21.15.27:0/4133240408 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7b6c00bbf0 con 0x7f7b80071ea0 2024-04-16T05:52:10.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.578+0000 7f7b77400700 1 --2- 172.21.15.27:0/4133240408 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7b6406ec00 0x7f7b640710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:10.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.578+0000 7f7b77400700 1 -- 172.21.15.27:0/4133240408 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f7b6c08fcf0 con 0x7f7b80071ea0 2024-04-16T05:52:10.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.578+0000 7f7b7f400700 1 --2- 172.21.15.27:0/4133240408 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7b6406ec00 0x7f7b640710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:10.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.579+0000 7f7b7f400700 1 --2- 172.21.15.27:0/4133240408 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7b6406ec00 0x7f7b640710b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7b80071be0 tx=0x7f7b7000b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:10.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.583+0000 7f7b77400700 1 -- 172.21.15.27:0/4133240408 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7b6c05c430 con 0x7f7b80071ea0 2024-04-16T05:52:10.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:10 smithi122 ceph-mon[80930]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:52:10.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:10 smithi122 ceph-mon[80930]: pgmap v242: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:10.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:10 smithi122 ceph-mon[80930]: from='client.34504 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:52:10.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:10 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:10.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:10 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:10.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:10 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/3386252529' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:10.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:10 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/743008933' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:52:10.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:10 smithi027 ceph-mon[93635]: from='client.34500 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:52:10.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:10 smithi027 ceph-mon[93635]: pgmap v242: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:10.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:10 smithi027 ceph-mon[93635]: from='client.34504 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:52:10.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:10 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:10.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:10 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:10.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:10 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/3386252529' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:10.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:10 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/743008933' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:52:10.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.839+0000 7f7b86400700 1 -- 172.21.15.27:0/4133240408 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b800ff8d0 con 0x7f7b6406ec00 2024-04-16T05:52:10.839 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.841+0000 7f7b77400700 1 -- 172.21.15.27:0/4133240408 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+441 (secure 0 0 0) 0x7f7b800ff8d0 con 0x7f7b6406ec00 2024-04-16T05:52:10.840 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:52:10.840 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T05:52:10.840 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:52:10.840 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:52:10.840 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [ 2024-04-16T05:52:10.840 INFO:teuthology.orchestra.run.smithi027.stdout: "mds", 2024-04-16T05:52:10.840 INFO:teuthology.orchestra.run.smithi027.stdout: "mon", 2024-04-16T05:52:10.840 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-16T05:52:10.840 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr", 2024-04-16T05:52:10.840 INFO:teuthology.orchestra.run.smithi027.stdout: "osd" 2024-04-16T05:52:10.840 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-16T05:52:10.840 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "16/21 daemons upgraded", 2024-04-16T05:52:10.840 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Currently upgrading node-exporter daemons", 2024-04-16T05:52:10.840 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:52:10.841 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:52:10.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.844+0000 7f7b86400700 1 -- 172.21.15.27:0/4133240408 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7b6406ec00 msgr2=0x7f7b640710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:10.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.844+0000 7f7b86400700 1 --2- 172.21.15.27:0/4133240408 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7b6406ec00 0x7f7b640710b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7b80071be0 tx=0x7f7b7000b040 comp rx=0 tx=0).stop 2024-04-16T05:52:10.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.844+0000 7f7b86400700 1 -- 172.21.15.27:0/4133240408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b80071ea0 msgr2=0x7f7b80114520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:10.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.844+0000 7f7b86400700 1 --2- 172.21.15.27:0/4133240408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b80071ea0 0x7f7b80114520 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00b970 tx=0x7f7b6c009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:10.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.844+0000 7f7b86400700 1 -- 172.21.15.27:0/4133240408 shutdown_connections 2024-04-16T05:52:10.843 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.844+0000 7f7b86400700 1 --2- 172.21.15.27:0/4133240408 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7b6406ec00 0x7f7b640710b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:10.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.844+0000 7f7b86400700 1 --2- 172.21.15.27:0/4133240408 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7b800708f0 0x7f7b80113fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:10.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.844+0000 7f7b86400700 1 --2- 172.21.15.27:0/4133240408 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b80071ea0 0x7f7b80114520 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:10.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.844+0000 7f7b86400700 1 -- 172.21.15.27:0/4133240408 >> 172.21.15.27:0/4133240408 conn(0x7f7b800f8580 msgr2=0x7f7b800f9ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:10.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.844+0000 7f7b86400700 1 -- 172.21.15.27:0/4133240408 shutdown_connections 2024-04-16T05:52:10.844 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:10.845+0000 7f7b86400700 1 -- 172.21.15.27:0/4133240408 wait complete. 2024-04-16T05:52:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.005+0000 7f8cfa400700 1 -- 172.21.15.27:0/422580861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cf40fff20 msgr2=0x7f8cf4102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.005+0000 7f8cfa400700 1 --2- 172.21.15.27:0/422580861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cf40fff20 0x7f8cf4102340 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc004940 tx=0x7f8cdc009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.006+0000 7f8cfa400700 1 -- 172.21.15.27:0/422580861 shutdown_connections 2024-04-16T05:52:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.006+0000 7f8cfa400700 1 --2- 172.21.15.27:0/422580861 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cf40fff20 0x7f8cf4102340 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.006+0000 7f8cfa400700 1 --2- 172.21.15.27:0/422580861 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8cf4066f00 0x7f8cf40ff9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.006+0000 7f8cfa400700 1 -- 172.21.15.27:0/422580861 >> 172.21.15.27:0/422580861 conn(0x7f8cf40f7450 msgr2=0x7f8cf40f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.006+0000 7f8cfa400700 1 -- 172.21.15.27:0/422580861 shutdown_connections 2024-04-16T05:52:11.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.006+0000 7f8cfa400700 1 -- 172.21.15.27:0/422580861 wait complete. 2024-04-16T05:52:11.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.007+0000 7f8cfa400700 1 Processor -- start 2024-04-16T05:52:11.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.007+0000 7f8cfa400700 1 -- start start 2024-04-16T05:52:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.007+0000 7f8cfa400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cf4066f00 0x7f8cf40fd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.007+0000 7f8cfa400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8cf40fff20 0x7f8cf40fb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.008+0000 7f8cfa400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf40fd830 con 0x7f8cf4066f00 2024-04-16T05:52:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.008+0000 7f8cfa400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf40fbe20 con 0x7f8cf40fff20 2024-04-16T05:52:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.008+0000 7f8cf3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cf4066f00 0x7f8cf40fd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.008+0000 7f8cf3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cf4066f00 0x7f8cf40fd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49478/0 (socket says 172.21.15.27:49478) 2024-04-16T05:52:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.008+0000 7f8cf3400700 1 -- 172.21.15.27:0/2932300118 learned_addr learned my addr 172.21.15.27:0/2932300118 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:52:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.008+0000 7f8cebe00700 1 --2- 172.21.15.27:0/2932300118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8cf40fff20 0x7f8cf40fb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:11.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.008+0000 7f8cf3400700 1 -- 172.21.15.27:0/2932300118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8cf40fff20 msgr2=0x7f8cf40fb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:11.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.008+0000 7f8cf3400700 1 --2- 172.21.15.27:0/2932300118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8cf40fff20 0x7f8cf40fb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:11.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.008+0000 7f8cf3400700 1 -- 172.21.15.27:0/2932300118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cdc005040 con 0x7f8cf4066f00 2024-04-16T05:52:11.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.008+0000 7f8cebe00700 1 --2- 172.21.15.27:0/2932300118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8cf40fff20 0x7f8cf40fb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T05:52:11.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.009+0000 7f8cf3400700 1 --2- 172.21.15.27:0/2932300118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cf4066f00 0x7f8cf40fd230 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4002a50 tx=0x7f8ce4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:11.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.009+0000 7f8ceb400700 1 -- 172.21.15.27:0/2932300118 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ce4007cc0 con 0x7f8cf4066f00 2024-04-16T05:52:11.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.009+0000 7f8cfa400700 1 -- 172.21.15.27:0/2932300118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cf40fc0d0 con 0x7f8cf4066f00 2024-04-16T05:52:11.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.009+0000 7f8ceb400700 1 -- 172.21.15.27:0/2932300118 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ce400f040 con 0x7f8cf4066f00 2024-04-16T05:52:11.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.009+0000 7f8ceb400700 1 -- 172.21.15.27:0/2932300118 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ce400da90 con 0x7f8cf4066f00 2024-04-16T05:52:11.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.009+0000 7f8cfa400700 1 -- 172.21.15.27:0/2932300118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cf40fc690 con 0x7f8cf4066f00 2024-04-16T05:52:11.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.010+0000 7f8cfa400700 1 -- 172.21.15.27:0/2932300118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cf404c710 con 0x7f8cf4066f00 2024-04-16T05:52:11.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.011+0000 7f8ceb400700 1 -- 172.21.15.27:0/2932300118 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8ce4009810 con 0x7f8cf4066f00 2024-04-16T05:52:11.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.012+0000 7f8ceb400700 1 --2- 172.21.15.27:0/2932300118 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8cd806ec00 0x7f8cd80710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:11.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.013+0000 7f8ceb400700 1 -- 172.21.15.27:0/2932300118 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f8ce408e8c0 con 0x7f8cf4066f00 2024-04-16T05:52:11.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.013+0000 7f8cebe00700 1 --2- 172.21.15.27:0/2932300118 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8cd806ec00 0x7f8cd80710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:11.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.014+0000 7f8cebe00700 1 --2- 172.21.15.27:0/2932300118 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8cd806ec00 0x7f8cd80710b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc00b3b0 tx=0x7f8cdc009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:11.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.018+0000 7f8ceb400700 1 -- 172.21.15.27:0/2932300118 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8ce405af00 con 0x7f8cf4066f00 2024-04-16T05:52:11.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.344+0000 7f8cfa400700 1 -- 172.21.15.27:0/2932300118 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f8cf41058f0 con 0x7f8cf4066f00 2024-04-16T05:52:11.343 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:52:11.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.344+0000 7f8ceb400700 1 -- 172.21.15.27:0/2932300118 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f8ce405aa90 con 0x7f8cf4066f00 2024-04-16T05:52:11.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.347+0000 7f8cfa400700 1 -- 172.21.15.27:0/2932300118 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8cd806ec00 msgr2=0x7f8cd80710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:11.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.347+0000 7f8cfa400700 1 --2- 172.21.15.27:0/2932300118 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8cd806ec00 0x7f8cd80710b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc00b3b0 tx=0x7f8cdc009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:11.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.347+0000 7f8cfa400700 1 -- 172.21.15.27:0/2932300118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cf4066f00 msgr2=0x7f8cf40fd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.347+0000 7f8cfa400700 1 --2- 172.21.15.27:0/2932300118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cf4066f00 0x7f8cf40fd230 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8ce4002a50 tx=0x7f8ce4004600 comp rx=0 tx=0).stop 2024-04-16T05:52:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.347+0000 7f8cfa400700 1 -- 172.21.15.27:0/2932300118 shutdown_connections 2024-04-16T05:52:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.347+0000 7f8cfa400700 1 --2- 172.21.15.27:0/2932300118 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8cd806ec00 0x7f8cd80710b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.347+0000 7f8cfa400700 1 --2- 172.21.15.27:0/2932300118 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8cf4066f00 0x7f8cf40fd230 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.347+0000 7f8cfa400700 1 --2- 172.21.15.27:0/2932300118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8cf40fff20 0x7f8cf40fb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.347+0000 7f8cfa400700 1 -- 172.21.15.27:0/2932300118 >> 172.21.15.27:0/2932300118 conn(0x7f8cf40f7450 msgr2=0x7f8cf40f7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.348+0000 7f8cfa400700 1 -- 172.21.15.27:0/2932300118 shutdown_connections 2024-04-16T05:52:11.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:11.348+0000 7f8cfa400700 1 -- 172.21.15.27:0/2932300118 wait complete. 2024-04-16T05:52:11.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:11 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/2932300118' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:52:11.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:11 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/2932300118' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:52:12.750 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:12 smithi027 ceph-mon[93635]: from='client.34516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:52:12.750 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:12 smithi027 ceph-mon[93635]: pgmap v243: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:12.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:12 smithi122 ceph-mon[80930]: from='client.34516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:52:12.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:12 smithi122 ceph-mon[80930]: pgmap v243: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:13.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:13 smithi122 ceph-mon[80930]: Upgrade: Updating node-exporter.smithi122 (2/2) 2024-04-16T05:52:13.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:13 smithi122 ceph-mon[80930]: Deploying daemon node-exporter.smithi122 on smithi122 2024-04-16T05:52:13.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:13 smithi027 ceph-mon[93635]: Upgrade: Updating node-exporter.smithi122 (2/2) 2024-04-16T05:52:13.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:13 smithi027 ceph-mon[93635]: Deploying daemon node-exporter.smithi122 on smithi122 2024-04-16T05:52:14.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:14 smithi122 ceph-mon[80930]: pgmap v244: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:14.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:52:14.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:14 smithi027 ceph-mon[93635]: pgmap v244: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:14.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:52:16.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:16 smithi122 ceph-mon[80930]: pgmap v245: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:16.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:16 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:16.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:16 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:16.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:16 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:52:16.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:16 smithi027 ceph-mon[93635]: pgmap v245: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:16.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:16 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:16.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:16 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:16.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:16 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:52:18.545 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:18 smithi122 ceph-mon[80930]: pgmap v246: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:18.801 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:18 smithi027 ceph-mon[93635]: pgmap v246: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:20.923 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:20 smithi027 ceph-mon[93635]: pgmap v247: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:20.923 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:20 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:20.923 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:20 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:21.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:20 smithi122 ceph-mon[80930]: pgmap v247: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:21.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:20 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:21.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:20 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:22.857 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:22 smithi122 ceph-mon[80930]: pgmap v248: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:22.857 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:22 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:22.857 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:22 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:23.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:22 smithi027 ceph-mon[93635]: pgmap v248: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:23.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:22 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:23.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:22 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:24.209 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:23 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:24.209 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:23 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:24.209 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:23 smithi122 ceph-mon[80930]: pgmap v249: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:24.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:23 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:24.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:23 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:24.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:23 smithi027 ceph-mon[93635]: pgmap v249: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:25.678 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 646, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 113646 2024-04-16T05:52:25.679 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 646, success, outsize: 120 2024-04-16T05:52:25.699 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 648, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 113724 2024-04-16T05:52:25.699 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 648, success, outsize: 120 2024-04-16T05:52:25.724 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 650, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 113805 2024-04-16T05:52:25.725 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 650, success, outsize: 120 2024-04-16T05:52:25.751 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 652, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 113892 2024-04-16T05:52:25.752 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 652, success, outsize: 120 2024-04-16T05:52:25.779 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 654, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 113976 2024-04-16T05:52:25.779 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 654, success, outsize: 120 2024-04-16T05:52:25.803 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 656, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114054 2024-04-16T05:52:25.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 656, success, outsize: 120 2024-04-16T05:52:25.826 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 658, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114135 2024-04-16T05:52:25.826 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 658, success, outsize: 120 2024-04-16T05:52:25.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 660, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114222 2024-04-16T05:52:25.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 660, success, outsize: 120 2024-04-16T05:52:25.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 662, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114306 2024-04-16T05:52:25.868 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 662, success, outsize: 120 2024-04-16T05:52:25.886 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 664, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114384 2024-04-16T05:52:25.886 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 664, success, outsize: 120 2024-04-16T05:52:25.902 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 666, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114465 2024-04-16T05:52:25.902 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 666, success, outsize: 120 2024-04-16T05:52:25.920 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 668, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114552 2024-04-16T05:52:25.921 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 668, success, outsize: 120 2024-04-16T05:52:25.940 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 670, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114636 2024-04-16T05:52:25.940 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 670, success, outsize: 120 2024-04-16T05:52:25.958 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 672, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114714 2024-04-16T05:52:25.958 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 672, success, outsize: 120 2024-04-16T05:52:25.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 674, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114795 2024-04-16T05:52:25.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 674, success, outsize: 120 2024-04-16T05:52:25.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 676, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114882 2024-04-16T05:52:25.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 676, success, outsize: 120 2024-04-16T05:52:26.016 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 678, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 114966 2024-04-16T05:52:26.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 678, success, outsize: 120 2024-04-16T05:52:26.035 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 680, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115044 2024-04-16T05:52:26.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 680, success, outsize: 120 2024-04-16T05:52:26.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 682, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115125 2024-04-16T05:52:26.057 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 682, success, outsize: 120 2024-04-16T05:52:26.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 684, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115212 2024-04-16T05:52:26.079 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 684, success, outsize: 120 2024-04-16T05:52:26.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 686, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115296 2024-04-16T05:52:26.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 686, success, outsize: 120 2024-04-16T05:52:26.118 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 688, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115374 2024-04-16T05:52:26.119 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 688, success, outsize: 120 2024-04-16T05:52:26.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 690, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115455 2024-04-16T05:52:26.139 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 690, success, outsize: 120 2024-04-16T05:52:26.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 692, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115542 2024-04-16T05:52:26.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 692, success, outsize: 120 2024-04-16T05:52:26.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 694, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115626 2024-04-16T05:52:26.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 694, success, outsize: 120 2024-04-16T05:52:26.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 696, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115704 2024-04-16T05:52:26.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 696, success, outsize: 120 2024-04-16T05:52:26.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 698, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115785 2024-04-16T05:52:26.216 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 698, success, outsize: 120 2024-04-16T05:52:26.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 700, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115872 2024-04-16T05:52:26.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr: unique: 700, success, outsize: 120 2024-04-16T05:52:26.400 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:26 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:26.400 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:26 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:26.400 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:26 smithi027 ceph-mon[93635]: pgmap v250: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:26.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:26 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:26.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:26 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:26.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:26 smithi122 ceph-mon[80930]: pgmap v250: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:28.439 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:28 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:28.439 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:28 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:28.439 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:28 smithi027 ceph-mon[93635]: pgmap v251: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:28.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:28 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:28.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:28 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:28.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:28 smithi122 ceph-mon[80930]: pgmap v251: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:29.469 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 478, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129752 2024-04-16T05:52:29.470 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 478, success, outsize: 120 2024-04-16T05:52:29.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 480, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129840 2024-04-16T05:52:29.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 480, success, outsize: 120 2024-04-16T05:52:29.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:52:29.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 482, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 129931 2024-04-16T05:52:29.539 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 482, success, outsize: 120 2024-04-16T05:52:29.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:52:29.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 484, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130028 2024-04-16T05:52:29.572 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 484, success, outsize: 120 2024-04-16T05:52:29.601 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 486, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130122 2024-04-16T05:52:29.601 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 486, success, outsize: 120 2024-04-16T05:52:29.626 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 488, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130210 2024-04-16T05:52:29.627 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 488, success, outsize: 120 2024-04-16T05:52:29.653 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 490, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130301 2024-04-16T05:52:29.654 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 490, success, outsize: 120 2024-04-16T05:52:29.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 492, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130398 2024-04-16T05:52:29.682 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 492, success, outsize: 120 2024-04-16T05:52:29.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 494, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130492 2024-04-16T05:52:29.709 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 494, success, outsize: 120 2024-04-16T05:52:29.734 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 496, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130580 2024-04-16T05:52:29.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 496, success, outsize: 120 2024-04-16T05:52:29.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 498, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130671 2024-04-16T05:52:29.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 498, success, outsize: 120 2024-04-16T05:52:29.795 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 500, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130768 2024-04-16T05:52:29.796 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 500, success, outsize: 120 2024-04-16T05:52:29.827 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 502, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130862 2024-04-16T05:52:29.827 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 502, success, outsize: 120 2024-04-16T05:52:29.858 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 504, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 130950 2024-04-16T05:52:29.859 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 504, success, outsize: 120 2024-04-16T05:52:29.884 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 506, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 131041 2024-04-16T05:52:29.884 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 506, success, outsize: 120 2024-04-16T05:52:29.909 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 508, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 131138 2024-04-16T05:52:29.910 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 508, success, outsize: 120 2024-04-16T05:52:29.936 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 510, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 131232 2024-04-16T05:52:29.936 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 510, success, outsize: 120 2024-04-16T05:52:29.962 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 512, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 131320 2024-04-16T05:52:29.962 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 512, success, outsize: 120 2024-04-16T05:52:29.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 514, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 131411 2024-04-16T05:52:29.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 514, success, outsize: 120 2024-04-16T05:52:30.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 516, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 131508 2024-04-16T05:52:30.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 516, success, outsize: 120 2024-04-16T05:52:30.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 518, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 131602 2024-04-16T05:52:30.048 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 518, success, outsize: 120 2024-04-16T05:52:30.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 520, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 131690 2024-04-16T05:52:30.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 520, success, outsize: 120 2024-04-16T05:52:30.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 522, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 131781 2024-04-16T05:52:30.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 522, success, outsize: 120 2024-04-16T05:52:30.134 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 524, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 131878 2024-04-16T05:52:30.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 524, success, outsize: 120 2024-04-16T05:52:30.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 526, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 131972 2024-04-16T05:52:30.162 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 526, success, outsize: 120 2024-04-16T05:52:30.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 528, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 132060 2024-04-16T05:52:30.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 528, success, outsize: 120 2024-04-16T05:52:30.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 530, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 132151 2024-04-16T05:52:30.211 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 530, success, outsize: 120 2024-04-16T05:52:30.234 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 532, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 132248 2024-04-16T05:52:30.235 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 532, success, outsize: 120 2024-04-16T05:52:30.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:30 smithi122 ceph-mon[80930]: pgmap v252: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:30.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:30 smithi027 ceph-mon[93635]: pgmap v252: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:32.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:32 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:32.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:32 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:32.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:32 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:52:32.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:32 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:52:32.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:32 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:32.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:32 smithi122 ceph-mon[80930]: pgmap v253: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:32.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:32 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:52:32.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:32 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:32 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:32 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:32 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:32 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:32 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:32 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:32 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:32 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:32 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:32 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:32 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:32 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:32 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:32.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:32 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:32.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:32 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:52:32.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:32 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:52:32.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:32 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:32.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:32 smithi027 ceph-mon[93635]: pgmap v253: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:32.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:32 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:52:32.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:32 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:32 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:32 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:32 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:32 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:32 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:32 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:32 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:32 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:32 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:32 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.559 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:32 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:32.559 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:32 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:34.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:34 smithi122 ceph-mon[80930]: pgmap v254: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:34.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:34 smithi027 ceph-mon[93635]: pgmap v254: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:35.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:35 smithi027 ceph-mon[93635]: Upgrade: Updating prometheus.smithi027 2024-04-16T05:52:35.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:35 smithi027 ceph-mon[93635]: Deploying daemon prometheus.smithi027 on smithi027 2024-04-16T05:52:35.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:35 smithi122 ceph-mon[80930]: Upgrade: Updating prometheus.smithi027 2024-04-16T05:52:35.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:35 smithi122 ceph-mon[80930]: Deploying daemon prometheus.smithi027 on smithi027 2024-04-16T05:52:36.757 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:36 smithi027 ceph-mon[93635]: pgmap v255: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:36.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:36 smithi122 ceph-mon[80930]: pgmap v255: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:38.715 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:38 smithi027 ceph-mon[93635]: pgmap v256: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:38.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:38 smithi122 ceph-mon[80930]: pgmap v256: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:40.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:40 smithi122 ceph-mon[80930]: pgmap v257: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:40 smithi027 ceph-mon[93635]: pgmap v257: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:41.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.519+0000 7fd66e000700 1 -- 172.21.15.27:0/458967750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6680fcef0 msgr2=0x7fd6680fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.519+0000 7fd66e000700 1 --2- 172.21.15.27:0/458967750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6680fcef0 0x7fd6680fd320 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd658004940 tx=0x7fd658009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.519+0000 7fd66e000700 1 -- 172.21.15.27:0/458967750 shutdown_connections 2024-04-16T05:52:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.519+0000 7fd66e000700 1 --2- 172.21.15.27:0/458967750 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6680fe0e0 0x7fd6680fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.519+0000 7fd66e000700 1 --2- 172.21.15.27:0/458967750 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6680fcef0 0x7fd6680fd320 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.519+0000 7fd66e000700 1 -- 172.21.15.27:0/458967750 >> 172.21.15.27:0/458967750 conn(0x7fd6680f8530 msgr2=0x7fd6680fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.520+0000 7fd66e000700 1 -- 172.21.15.27:0/458967750 shutdown_connections 2024-04-16T05:52:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.520+0000 7fd66e000700 1 -- 172.21.15.27:0/458967750 wait complete. 2024-04-16T05:52:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.521+0000 7fd66e000700 1 Processor -- start 2024-04-16T05:52:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.521+0000 7fd66e000700 1 -- start start 2024-04-16T05:52:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.521+0000 7fd66e000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6680fcef0 0x7fd66810da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.521+0000 7fd66e000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6680fe0e0 0x7fd66810dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:41.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.521+0000 7fd66e000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66810e5c0 con 0x7fd6680fcef0 2024-04-16T05:52:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.521+0000 7fd66e000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66810ee10 con 0x7fd6680fe0e0 2024-04-16T05:52:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.522+0000 7fd667400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6680fcef0 0x7fd66810da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.522+0000 7fd667400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6680fcef0 0x7fd66810da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36374/0 (socket says 172.21.15.27:36374) 2024-04-16T05:52:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.522+0000 7fd667400700 1 -- 172.21.15.27:0/1751125959 learned_addr learned my addr 172.21.15.27:0/1751125959 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:52:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.522+0000 7fd666a00700 1 --2- 172.21.15.27:0/1751125959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6680fe0e0 0x7fd66810dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.522+0000 7fd667400700 1 -- 172.21.15.27:0/1751125959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6680fe0e0 msgr2=0x7fd66810dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.522+0000 7fd667400700 1 --2- 172.21.15.27:0/1751125959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6680fe0e0 0x7fd66810dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.522+0000 7fd667400700 1 -- 172.21.15.27:0/1751125959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd658005040 con 0x7fd6680fcef0 2024-04-16T05:52:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.522+0000 7fd666a00700 1 --2- 172.21.15.27:0/1751125959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6680fe0e0 0x7fd66810dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:52:41.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.523+0000 7fd667400700 1 --2- 172.21.15.27:0/1751125959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6680fcef0 0x7fd66810da80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd658000c00 tx=0x7fd658009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.523+0000 7fd65f400700 1 -- 172.21.15.27:0/1751125959 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd65800e040 con 0x7fd6680fcef0 2024-04-16T05:52:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.523+0000 7fd65f400700 1 -- 172.21.15.27:0/1751125959 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd65800bcc0 con 0x7fd6680fcef0 2024-04-16T05:52:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.523+0000 7fd66e000700 1 -- 172.21.15.27:0/1751125959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd66810f090 con 0x7fd6680fcef0 2024-04-16T05:52:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.523+0000 7fd65f400700 1 -- 172.21.15.27:0/1751125959 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd65801c630 con 0x7fd6680fcef0 2024-04-16T05:52:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.523+0000 7fd66e000700 1 -- 172.21.15.27:0/1751125959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd66810f5d0 con 0x7fd6680fcef0 2024-04-16T05:52:41.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.525+0000 7fd66e000700 1 -- 172.21.15.27:0/1751125959 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd66805f5f0 con 0x7fd6680fcef0 2024-04-16T05:52:41.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.526+0000 7fd65f400700 1 -- 172.21.15.27:0/1751125959 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd658013980 con 0x7fd6680fcef0 2024-04-16T05:52:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.527+0000 7fd65f400700 1 --2- 172.21.15.27:0/1751125959 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd64c06ec00 0x7fd64c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.527+0000 7fd65f400700 1 -- 172.21.15.27:0/1751125959 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7fd65808fe40 con 0x7fd6680fcef0 2024-04-16T05:52:41.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.527+0000 7fd666a00700 1 --2- 172.21.15.27:0/1751125959 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd64c06ec00 0x7fd64c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:41.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.528+0000 7fd666a00700 1 --2- 172.21.15.27:0/1751125959 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd64c06ec00 0x7fd64c0710b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd654009fd0 tx=0x7fd65400b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:41.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.533+0000 7fd65f400700 1 -- 172.21.15.27:0/1751125959 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd65805c5a0 con 0x7fd6680fcef0 2024-04-16T05:52:41.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.794+0000 7fd66e000700 1 -- 172.21.15.27:0/1751125959 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd66810f920 con 0x7fd64c06ec00 2024-04-16T05:52:41.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.795+0000 7fd65f400700 1 -- 172.21.15.27:0/1751125959 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+438 (secure 0 0 0) 0x7fd66810f920 con 0x7fd64c06ec00 2024-04-16T05:52:41.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.798+0000 7fd66e000700 1 -- 172.21.15.27:0/1751125959 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd64c06ec00 msgr2=0x7fd64c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:41.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.798+0000 7fd66e000700 1 --2- 172.21.15.27:0/1751125959 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd64c06ec00 0x7fd64c0710b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd654009fd0 tx=0x7fd65400b040 comp rx=0 tx=0).stop 2024-04-16T05:52:41.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.798+0000 7fd66e000700 1 -- 172.21.15.27:0/1751125959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6680fcef0 msgr2=0x7fd66810da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:41.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.798+0000 7fd66e000700 1 --2- 172.21.15.27:0/1751125959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6680fcef0 0x7fd66810da80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd658000c00 tx=0x7fd658009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:41.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.799+0000 7fd66e000700 1 -- 172.21.15.27:0/1751125959 shutdown_connections 2024-04-16T05:52:41.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.799+0000 7fd66e000700 1 --2- 172.21.15.27:0/1751125959 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd64c06ec00 0x7fd64c0710b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:41.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.799+0000 7fd66e000700 1 --2- 172.21.15.27:0/1751125959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd6680fcef0 0x7fd66810da80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:41.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.799+0000 7fd66e000700 1 --2- 172.21.15.27:0/1751125959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd6680fe0e0 0x7fd66810dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:41.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.799+0000 7fd66e000700 1 -- 172.21.15.27:0/1751125959 >> 172.21.15.27:0/1751125959 conn(0x7fd6680f8530 msgr2=0x7fd6680f9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:41.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.799+0000 7fd66e000700 1 -- 172.21.15.27:0/1751125959 shutdown_connections 2024-04-16T05:52:41.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.799+0000 7fd66e000700 1 -- 172.21.15.27:0/1751125959 wait complete. 2024-04-16T05:52:41.815 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:52:41.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.961+0000 7f91c4a00700 1 -- 172.21.15.27:0/1119907024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91c00fff70 msgr2=0x7f91c0102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:41.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.961+0000 7f91c4a00700 1 --2- 172.21.15.27:0/1119907024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91c00fff70 0x7f91c0102390 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f91ac004940 tx=0x7f91ac009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:41.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.961+0000 7f91c4a00700 1 -- 172.21.15.27:0/1119907024 shutdown_connections 2024-04-16T05:52:41.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.961+0000 7f91c4a00700 1 --2- 172.21.15.27:0/1119907024 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91c00fff70 0x7f91c0102390 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:41.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.961+0000 7f91c4a00700 1 --2- 172.21.15.27:0/1119907024 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91c00662f0 0x7f91c00ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:41.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.961+0000 7f91c4a00700 1 -- 172.21.15.27:0/1119907024 >> 172.21.15.27:0/1119907024 conn(0x7f91c00f74c0 msgr2=0x7f91c00f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:41.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.961+0000 7f91c4a00700 1 -- 172.21.15.27:0/1119907024 shutdown_connections 2024-04-16T05:52:41.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.961+0000 7f91c4a00700 1 -- 172.21.15.27:0/1119907024 wait complete. 2024-04-16T05:52:41.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.962+0000 7f91c4a00700 1 Processor -- start 2024-04-16T05:52:41.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.963+0000 7f91c4a00700 1 -- start start 2024-04-16T05:52:41.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.963+0000 7f91c4a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91c00662f0 0x7f91c010da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:41.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.963+0000 7f91c4a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91c00fff70 0x7f91c010dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:41.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.963+0000 7f91c4a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c010e5a0 con 0x7f91c00662f0 2024-04-16T05:52:41.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.963+0000 7f91c4a00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c010edf0 con 0x7f91c00fff70 2024-04-16T05:52:41.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.963+0000 7f91bde00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91c00662f0 0x7f91c010da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:41.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.963+0000 7f91bde00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91c00662f0 0x7f91c010da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36378/0 (socket says 172.21.15.27:36378) 2024-04-16T05:52:41.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.963+0000 7f91bde00700 1 -- 172.21.15.27:0/2404164425 learned_addr learned my addr 172.21.15.27:0/2404164425 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:52:41.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.964+0000 7f91bd400700 1 --2- 172.21.15.27:0/2404164425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91c00fff70 0x7f91c010dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:41.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.964+0000 7f91bde00700 1 -- 172.21.15.27:0/2404164425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91c00fff70 msgr2=0x7f91c010dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:41.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.964+0000 7f91bde00700 1 --2- 172.21.15.27:0/2404164425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91c00fff70 0x7f91c010dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:41.963 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.964+0000 7f91bde00700 1 -- 172.21.15.27:0/2404164425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ac005040 con 0x7f91c00662f0 2024-04-16T05:52:41.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.964+0000 7f91bde00700 1 --2- 172.21.15.27:0/2404164425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91c00662f0 0x7f91c010da60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f91b0002a50 tx=0x7f91b0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:41.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.964+0000 7f91b6000700 1 -- 172.21.15.27:0/2404164425 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91b0007cc0 con 0x7f91c00662f0 2024-04-16T05:52:41.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.965+0000 7f91b6000700 1 -- 172.21.15.27:0/2404164425 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f91b000f040 con 0x7f91c00662f0 2024-04-16T05:52:41.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.965+0000 7f91b6000700 1 -- 172.21.15.27:0/2404164425 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91b000daf0 con 0x7f91c00662f0 2024-04-16T05:52:41.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.965+0000 7f91c4a00700 1 -- 172.21.15.27:0/2404164425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c010f0d0 con 0x7f91c00662f0 2024-04-16T05:52:41.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.965+0000 7f91c4a00700 1 -- 172.21.15.27:0/2404164425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c010f610 con 0x7f91c00662f0 2024-04-16T05:52:41.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.966+0000 7f91c4a00700 1 -- 172.21.15.27:0/2404164425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c005f5f0 con 0x7f91c00662f0 2024-04-16T05:52:41.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.967+0000 7f91b6000700 1 -- 172.21.15.27:0/2404164425 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f91b0009810 con 0x7f91c00662f0 2024-04-16T05:52:41.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.968+0000 7f91b6000700 1 --2- 172.21.15.27:0/2404164425 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f91a406ec00 0x7f91a40710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:41.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.968+0000 7f91b6000700 1 -- 172.21.15.27:0/2404164425 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f91b008e750 con 0x7f91c00662f0 2024-04-16T05:52:41.967 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.968+0000 7f91bd400700 1 --2- 172.21.15.27:0/2404164425 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f91a406ec00 0x7f91a40710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:41.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.969+0000 7f91bd400700 1 --2- 172.21.15.27:0/2404164425 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f91a406ec00 0x7f91a40710b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f91ac00b3b0 tx=0x7f91ac009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:41.971 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:41.973+0000 7f91b6000700 1 -- 172.21.15.27:0/2404164425 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f91b005aeb0 con 0x7f91c00662f0 2024-04-16T05:52:42.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.273+0000 7f91c4a00700 1 -- 172.21.15.27:0/2404164425 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f91c010f9e0 con 0x7f91a406ec00 2024-04-16T05:52:42.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.275+0000 7f91b6000700 1 -- 172.21.15.27:0/2404164425 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+438 (secure 0 0 0) 0x7f91c010f9e0 con 0x7f91a406ec00 2024-04-16T05:52:42.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.278+0000 7f91c4a00700 1 -- 172.21.15.27:0/2404164425 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f91a406ec00 msgr2=0x7f91a40710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:42.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.278+0000 7f91c4a00700 1 --2- 172.21.15.27:0/2404164425 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f91a406ec00 0x7f91a40710b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f91ac00b3b0 tx=0x7f91ac009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:42.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.278+0000 7f91c4a00700 1 -- 172.21.15.27:0/2404164425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91c00662f0 msgr2=0x7f91c010da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:42.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.278+0000 7f91c4a00700 1 --2- 172.21.15.27:0/2404164425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91c00662f0 0x7f91c010da60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f91b0002a50 tx=0x7f91b0004600 comp rx=0 tx=0).stop 2024-04-16T05:52:42.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.279+0000 7f91c4a00700 1 -- 172.21.15.27:0/2404164425 shutdown_connections 2024-04-16T05:52:42.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.279+0000 7f91c4a00700 1 --2- 172.21.15.27:0/2404164425 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f91a406ec00 0x7f91a40710b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:42.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.279+0000 7f91c4a00700 1 --2- 172.21.15.27:0/2404164425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91c00662f0 0x7f91c010da60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:42.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.279+0000 7f91c4a00700 1 --2- 172.21.15.27:0/2404164425 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91c00fff70 0x7f91c010dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:42.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.279+0000 7f91c4a00700 1 -- 172.21.15.27:0/2404164425 >> 172.21.15.27:0/2404164425 conn(0x7f91c00f74c0 msgr2=0x7f91c00f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:42.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.279+0000 7f91c4a00700 1 -- 172.21.15.27:0/2404164425 shutdown_connections 2024-04-16T05:52:42.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.279+0000 7f91c4a00700 1 -- 172.21.15.27:0/2404164425 wait complete. 2024-04-16T05:52:42.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.450+0000 7f860c200700 1 -- 172.21.15.27:0/681334441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86040fe0e0 msgr2=0x7f86040fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:42.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.450+0000 7f860c200700 1 --2- 172.21.15.27:0/681334441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86040fe0e0 0x7f86040fe550 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f85f0004940 tx=0x7f85f0009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:42.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.450+0000 7f860c200700 1 -- 172.21.15.27:0/681334441 shutdown_connections 2024-04-16T05:52:42.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.450+0000 7f860c200700 1 --2- 172.21.15.27:0/681334441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86040fe0e0 0x7f86040fe550 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:42.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.450+0000 7f860c200700 1 --2- 172.21.15.27:0/681334441 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86040fcef0 0x7f86040fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:42.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.450+0000 7f860c200700 1 -- 172.21.15.27:0/681334441 >> 172.21.15.27:0/681334441 conn(0x7f86040f8530 msgr2=0x7f86040fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:42.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.450+0000 7f860c200700 1 -- 172.21.15.27:0/681334441 shutdown_connections 2024-04-16T05:52:42.450 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.450+0000 7f860c200700 1 -- 172.21.15.27:0/681334441 wait complete. 2024-04-16T05:52:42.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.452+0000 7f860c200700 1 Processor -- start 2024-04-16T05:52:42.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.452+0000 7f860c200700 1 -- start start 2024-04-16T05:52:42.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.453+0000 7f860c200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86040fcef0 0x7f8604071df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:42.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.453+0000 7f860c200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86040fe0e0 0x7f8604070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:42.451 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.453+0000 7f860c200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86040708a0 con 0x7f86040fe0e0 2024-04-16T05:52:42.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.453+0000 7f860c200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86040709e0 con 0x7f86040fcef0 2024-04-16T05:52:42.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.453+0000 7f8608e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86040fe0e0 0x7f8604070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:42.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.453+0000 7f8608e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86040fe0e0 0x7f8604070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36396/0 (socket says 172.21.15.27:36396) 2024-04-16T05:52:42.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.453+0000 7f8608e00700 1 -- 172.21.15.27:0/2129248654 learned_addr learned my addr 172.21.15.27:0/2129248654 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:52:42.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.453+0000 7f8609800700 1 --2- 172.21.15.27:0/2129248654 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86040fcef0 0x7f8604071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:42.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.454+0000 7f8608e00700 1 -- 172.21.15.27:0/2129248654 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86040fcef0 msgr2=0x7f8604071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:42.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.454+0000 7f8608e00700 1 --2- 172.21.15.27:0/2129248654 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86040fcef0 0x7f8604071df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:42.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.454+0000 7f8608e00700 1 -- 172.21.15.27:0/2129248654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f0005040 con 0x7f86040fe0e0 2024-04-16T05:52:42.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.454+0000 7f8608e00700 1 --2- 172.21.15.27:0/2129248654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86040fe0e0 0x7f8604070310 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f85f000b3b0 tx=0x7f85f0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:42.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.455+0000 7f8601600700 1 -- 172.21.15.27:0/2129248654 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85f000e040 con 0x7f86040fe0e0 2024-04-16T05:52:42.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.455+0000 7f860c200700 1 -- 172.21.15.27:0/2129248654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8604070c60 con 0x7f86040fe0e0 2024-04-16T05:52:42.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.455+0000 7f8601600700 1 -- 172.21.15.27:0/2129248654 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f85f0004290 con 0x7f86040fe0e0 2024-04-16T05:52:42.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.455+0000 7f8601600700 1 -- 172.21.15.27:0/2129248654 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85f001c630 con 0x7f86040fe0e0 2024-04-16T05:52:42.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.455+0000 7f860c200700 1 -- 172.21.15.27:0/2129248654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8604115ac0 con 0x7f86040fe0e0 2024-04-16T05:52:42.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.456+0000 7f860c200700 1 -- 172.21.15.27:0/2129248654 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e40052e0 con 0x7f86040fe0e0 2024-04-16T05:52:42.457 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.458+0000 7f8601600700 1 -- 172.21.15.27:0/2129248654 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f85f000bbf0 con 0x7f86040fe0e0 2024-04-16T05:52:42.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.460+0000 7f8601600700 1 --2- 172.21.15.27:0/2129248654 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f85ec06ec10 0x7f85ec0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:42.458 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.460+0000 7f8601600700 1 -- 172.21.15.27:0/2129248654 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f85f008fda0 con 0x7f86040fe0e0 2024-04-16T05:52:42.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.460+0000 7f8609800700 1 --2- 172.21.15.27:0/2129248654 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f85ec06ec10 0x7f85ec0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:42.459 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.461+0000 7f8609800700 1 --2- 172.21.15.27:0/2129248654 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f85ec06ec10 0x7f85ec0710c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8604071b80 tx=0x7f85f800b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:42.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.466+0000 7f8601600700 1 -- 172.21.15.27:0/2129248654 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f85f00589f0 con 0x7f86040fe0e0 2024-04-16T05:52:42.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.713+0000 7f860c200700 1 -- 172.21.15.27:0/2129248654 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f85e4000bf0 con 0x7f85ec06ec10 2024-04-16T05:52:42.713 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:42 smithi027 ceph-mon[93635]: pgmap v258: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:42.713 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:42 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:42.713 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:42 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:42.713 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:42 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:52:42.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.743+0000 7f8601600700 1 -- 172.21.15.27:0/2129248654 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f85e4000bf0 con 0x7f85ec06ec10 2024-04-16T05:52:42.743 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:52:42.743 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (9m) 19s ago 19m 13.5M - 0.25.0 c8568f914cd2 4ed63a8a6b33 2024-04-16T05:52:42.743 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (5m) 19s ago 20m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 4aa1f0006685 2024-04-16T05:52:42.743 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (5m) 23s ago 18m 6656k - 19.0.0-2884-g218a6914 a4e4811815e4 1c1912507d56 2024-04-16T05:52:42.744 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (8m) 19s ago 19m 75.0M - 9.4.7 954c08fa6188 f7ee2a1afd12 2024-04-16T05:52:42.744 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (107s) 19s ago 12m 11.5M - 19.0.0-2884-g218a6914 a4e4811815e4 b3c2178c537e 2024-04-16T05:52:42.744 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (2m) 19s ago 12m 15.9M - 19.0.0-2884-g218a6914 a4e4811815e4 14b0c83f2891 2024-04-16T05:52:42.744 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (76s) 23s ago 12m 11.2M - 19.0.0-2884-g218a6914 a4e4811815e4 8ba95adfaaef 2024-04-16T05:52:42.744 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (52s) 23s ago 12m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 c60e9ca0f1c8 2024-04-16T05:52:42.744 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:8443,9283,8765 running (8m) 19s ago 21m 511M - 19.0.0-2884-g218a6914 a4e4811815e4 b69a0b08e12b 2024-04-16T05:52:42.744 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283,8765 running (7m) 23s ago 18m 454M - 19.0.0-2884-g218a6914 a4e4811815e4 32e27bcd7e42 2024-04-16T05:52:42.744 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (7m) 19s ago 21m 53.6M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 299ca337e4ca 2024-04-16T05:52:42.744 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (6m) 23s ago 18m 40.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5db5f3037ed0 2024-04-16T05:52:42.745 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (33s) 19s ago 19m 6883k - 1.5.0 0da6a335fe13 f3382c6beacb 2024-04-16T05:52:42.745 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (27s) 23s ago 18m 3078k - 1.5.0 0da6a335fe13 1f6817821f3a 2024-04-16T05:52:42.745 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (5m) 19s ago 17m 46.9M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 cef03b67bc5a 2024-04-16T05:52:42.745 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (4m) 19s ago 16m 45.8M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3a6ab1fc6e79 2024-04-16T05:52:42.745 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (4m) 19s ago 15m 44.4M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3524731d7277 2024-04-16T05:52:42.746 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (3m) 23s ago 15m 44.7M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 897d3579f929 2024-04-16T05:52:42.746 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (3m) 23s ago 14m 42.2M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 fa97758ff7c5 2024-04-16T05:52:42.746 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (2m) 23s ago 14m 49.6M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 fd12260a820d 2024-04-16T05:52:42.746 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 starting - - - - 2024-04-16T05:52:42.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.746+0000 7f860c200700 1 -- 172.21.15.27:0/2129248654 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f85ec06ec10 msgr2=0x7f85ec0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:42.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.746+0000 7f860c200700 1 --2- 172.21.15.27:0/2129248654 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f85ec06ec10 0x7f85ec0710c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8604071b80 tx=0x7f85f800b040 comp rx=0 tx=0).stop 2024-04-16T05:52:42.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.746+0000 7f860c200700 1 -- 172.21.15.27:0/2129248654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86040fe0e0 msgr2=0x7f8604070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:42.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.746+0000 7f860c200700 1 --2- 172.21.15.27:0/2129248654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86040fe0e0 0x7f8604070310 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f85f000b3b0 tx=0x7f85f0009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:42.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.746+0000 7f860c200700 1 -- 172.21.15.27:0/2129248654 shutdown_connections 2024-04-16T05:52:42.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.746+0000 7f860c200700 1 --2- 172.21.15.27:0/2129248654 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f85ec06ec10 0x7f85ec0710c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:42.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.746+0000 7f860c200700 1 --2- 172.21.15.27:0/2129248654 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f86040fcef0 0x7f8604071df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:42.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.746+0000 7f860c200700 1 --2- 172.21.15.27:0/2129248654 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86040fe0e0 0x7f8604070310 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:42.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.746+0000 7f860c200700 1 -- 172.21.15.27:0/2129248654 >> 172.21.15.27:0/2129248654 conn(0x7f86040f8530 msgr2=0x7f86040f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:42.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.747+0000 7f860c200700 1 -- 172.21.15.27:0/2129248654 shutdown_connections 2024-04-16T05:52:42.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.747+0000 7f860c200700 1 -- 172.21.15.27:0/2129248654 wait complete. 2024-04-16T05:52:42.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:42 smithi122 ceph-mon[80930]: pgmap v258: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:42.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:42 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:42.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:42 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:42.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:42 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:52:42.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.909+0000 7ff53c800700 1 -- 172.21.15.27:0/2880211241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5340662f0 msgr2=0x7ff5340ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:42.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.909+0000 7ff53c800700 1 --2- 172.21.15.27:0/2880211241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5340662f0 0x7ff5340ffa30 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff528004940 tx=0x7ff528009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:42.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.910+0000 7ff53c800700 1 -- 172.21.15.27:0/2880211241 shutdown_connections 2024-04-16T05:52:42.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.910+0000 7ff53c800700 1 --2- 172.21.15.27:0/2880211241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5340fff70 0x7ff534102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:42.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.910+0000 7ff53c800700 1 --2- 172.21.15.27:0/2880211241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5340662f0 0x7ff5340ffa30 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:42.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.910+0000 7ff53c800700 1 -- 172.21.15.27:0/2880211241 >> 172.21.15.27:0/2880211241 conn(0x7ff5340f74c0 msgr2=0x7ff5340f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:42.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.910+0000 7ff53c800700 1 -- 172.21.15.27:0/2880211241 shutdown_connections 2024-04-16T05:52:42.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.910+0000 7ff53c800700 1 -- 172.21.15.27:0/2880211241 wait complete. 2024-04-16T05:52:42.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.911+0000 7ff53c800700 1 Processor -- start 2024-04-16T05:52:42.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.911+0000 7ff53c800700 1 -- start start 2024-04-16T05:52:42.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.911+0000 7ff53c800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5340662f0 0x7ff53410fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:42.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.911+0000 7ff53c800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5340fff70 0x7ff5341101e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:42.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.911+0000 7ff53c800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff534110870 con 0x7ff5340fff70 2024-04-16T05:52:42.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.911+0000 7ff53c800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff534111010 con 0x7ff5340662f0 2024-04-16T05:52:42.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.912+0000 7ff539400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5340fff70 0x7ff5341101e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:42.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.912+0000 7ff539400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5340fff70 0x7ff5341101e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36406/0 (socket says 172.21.15.27:36406) 2024-04-16T05:52:42.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.912+0000 7ff539400700 1 -- 172.21.15.27:0/4080168671 learned_addr learned my addr 172.21.15.27:0/4080168671 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:52:42.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.912+0000 7ff539e00700 1 --2- 172.21.15.27:0/4080168671 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5340662f0 0x7ff53410fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:42.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.912+0000 7ff539400700 1 -- 172.21.15.27:0/4080168671 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5340662f0 msgr2=0x7ff53410fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:42.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.912+0000 7ff539400700 1 --2- 172.21.15.27:0/4080168671 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5340662f0 0x7ff53410fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:42.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.912+0000 7ff539400700 1 -- 172.21.15.27:0/4080168671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff528005040 con 0x7ff5340fff70 2024-04-16T05:52:42.912 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.913+0000 7ff539400700 1 --2- 172.21.15.27:0/4080168671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5340fff70 0x7ff5341101e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff52400b790 tx=0x7ff524009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:42.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.913+0000 7ff52e000700 1 -- 172.21.15.27:0/4080168671 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff52400f770 con 0x7ff5340fff70 2024-04-16T05:52:42.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.913+0000 7ff52e000700 1 -- 172.21.15.27:0/4080168671 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff52400fda0 con 0x7ff5340fff70 2024-04-16T05:52:42.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.913+0000 7ff53c800700 1 -- 172.21.15.27:0/4080168671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5341112f0 con 0x7ff5340fff70 2024-04-16T05:52:42.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.913+0000 7ff52e000700 1 -- 172.21.15.27:0/4080168671 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff52400d310 con 0x7ff5340fff70 2024-04-16T05:52:42.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.913+0000 7ff53c800700 1 -- 172.21.15.27:0/4080168671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5341118b0 con 0x7ff5340fff70 2024-04-16T05:52:42.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.915+0000 7ff53c800700 1 -- 172.21.15.27:0/4080168671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff53405f5f0 con 0x7ff5340fff70 2024-04-16T05:52:42.914 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.916+0000 7ff52e000700 1 -- 172.21.15.27:0/4080168671 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff524010420 con 0x7ff5340fff70 2024-04-16T05:52:42.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.916+0000 7ff52e000700 1 --2- 172.21.15.27:0/4080168671 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff51c06ec10 0x7ff51c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:42.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.917+0000 7ff52e000700 1 -- 172.21.15.27:0/4080168671 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7ff52408eac0 con 0x7ff5340fff70 2024-04-16T05:52:42.915 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.917+0000 7ff539e00700 1 --2- 172.21.15.27:0/4080168671 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff51c06ec10 0x7ff51c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:42.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.918+0000 7ff539e00700 1 --2- 172.21.15.27:0/4080168671 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff51c06ec10 0x7ff51c0710c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff52800b3b0 tx=0x7ff528009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:42.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:42.921+0000 7ff52e000700 1 -- 172.21.15.27:0/4080168671 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff52405b200 con 0x7ff5340fff70 2024-04-16T05:52:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.266+0000 7ff53c800700 1 -- 172.21.15.27:0/4080168671 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff534111b80 con 0x7ff5340fff70 2024-04-16T05:52:43.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.266+0000 7ff52e000700 1 -- 172.21.15.27:0/4080168671 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7ff52405ad90 con 0x7ff5340fff70 2024-04-16T05:52:43.265 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:52:43.266 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:52:43.266 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:52:43.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:52:43.266 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:52:43.266 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:52:43.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:52:43.266 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:52:43.266 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T05:52:43.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:52:43.266 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:52:43.266 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T05:52:43.266 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:52:43.266 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:52:43.266 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 14 2024-04-16T05:52:43.267 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:52:43.267 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:52:43.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.269+0000 7ff53c800700 1 -- 172.21.15.27:0/4080168671 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff51c06ec10 msgr2=0x7ff51c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:43.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.269+0000 7ff53c800700 1 --2- 172.21.15.27:0/4080168671 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff51c06ec10 0x7ff51c0710c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff52800b3b0 tx=0x7ff528009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:43.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.269+0000 7ff53c800700 1 -- 172.21.15.27:0/4080168671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5340fff70 msgr2=0x7ff5341101e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:43.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.269+0000 7ff53c800700 1 --2- 172.21.15.27:0/4080168671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5340fff70 0x7ff5341101e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff52400b790 tx=0x7ff524009250 comp rx=0 tx=0).stop 2024-04-16T05:52:43.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.269+0000 7ff53c800700 1 -- 172.21.15.27:0/4080168671 shutdown_connections 2024-04-16T05:52:43.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.269+0000 7ff53c800700 1 --2- 172.21.15.27:0/4080168671 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff51c06ec10 0x7ff51c0710c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:43.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.269+0000 7ff53c800700 1 --2- 172.21.15.27:0/4080168671 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff5340662f0 0x7ff53410fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:43.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.269+0000 7ff53c800700 1 --2- 172.21.15.27:0/4080168671 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff5340fff70 0x7ff5341101e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:43.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.269+0000 7ff53c800700 1 -- 172.21.15.27:0/4080168671 >> 172.21.15.27:0/4080168671 conn(0x7ff5340f74c0 msgr2=0x7ff5340fbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:43.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.270+0000 7ff53c800700 1 -- 172.21.15.27:0/4080168671 shutdown_connections 2024-04-16T05:52:43.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.270+0000 7ff53c800700 1 -- 172.21.15.27:0/4080168671 wait complete. 2024-04-16T05:52:43.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.420+0000 7f9126e00700 1 -- 172.21.15.27:0/2731729599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91200fd710 msgr2=0x7f91200fe070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:43.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.420+0000 7f9126e00700 1 --2- 172.21.15.27:0/2731729599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91200fd710 0x7f91200fe070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9108004940 tx=0x7f9108009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:43.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.420+0000 7f9126e00700 1 -- 172.21.15.27:0/2731729599 shutdown_connections 2024-04-16T05:52:43.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.420+0000 7f9126e00700 1 --2- 172.21.15.27:0/2731729599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91200fd710 0x7f91200fe070 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:43.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.421+0000 7f9126e00700 1 --2- 172.21.15.27:0/2731729599 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91200fcdc0 0x7f91200fd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:43.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.421+0000 7f9126e00700 1 -- 172.21.15.27:0/2731729599 >> 172.21.15.27:0/2731729599 conn(0x7f91200f8540 msgr2=0x7f91200fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:43.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.421+0000 7f9126e00700 1 -- 172.21.15.27:0/2731729599 shutdown_connections 2024-04-16T05:52:43.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.421+0000 7f9126e00700 1 -- 172.21.15.27:0/2731729599 wait complete. 2024-04-16T05:52:43.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.421+0000 7f9126e00700 1 Processor -- start 2024-04-16T05:52:43.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.422+0000 7f9126e00700 1 -- start start 2024-04-16T05:52:43.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.422+0000 7f9126e00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91200fcdc0 0x7f912010fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:43.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.422+0000 7f9126e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91200fd710 0x7f91201102c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:43.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.422+0000 7f9126e00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91201108c0 con 0x7f91200fd710 2024-04-16T05:52:43.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.422+0000 7f9126e00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9120111000 con 0x7f91200fcdc0 2024-04-16T05:52:43.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.422+0000 7f9117400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91200fd710 0x7f91201102c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:43.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.422+0000 7f9117400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91200fd710 0x7f91201102c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36426/0 (socket says 172.21.15.27:36426) 2024-04-16T05:52:43.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.422+0000 7f9117400700 1 -- 172.21.15.27:0/576539210 learned_addr learned my addr 172.21.15.27:0/576539210 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:52:43.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.423+0000 7f911fe00700 1 --2- 172.21.15.27:0/576539210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91200fcdc0 0x7f912010fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:43.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.423+0000 7f9117400700 1 -- 172.21.15.27:0/576539210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91200fcdc0 msgr2=0x7f912010fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:43.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.423+0000 7f9117400700 1 --2- 172.21.15.27:0/576539210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91200fcdc0 0x7f912010fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:43.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.423+0000 7f9117400700 1 -- 172.21.15.27:0/576539210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9108005040 con 0x7f91200fd710 2024-04-16T05:52:43.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.423+0000 7f9117400700 1 --2- 172.21.15.27:0/576539210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91200fd710 0x7f91201102c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f910800b970 tx=0x7f9108009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:43.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.423+0000 7f911cc00700 1 -- 172.21.15.27:0/576539210 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f910800e040 con 0x7f91200fd710 2024-04-16T05:52:43.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.423+0000 7f9126e00700 1 -- 172.21.15.27:0/576539210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9120111280 con 0x7f91200fd710 2024-04-16T05:52:43.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.423+0000 7f911cc00700 1 -- 172.21.15.27:0/576539210 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9108004290 con 0x7f91200fd710 2024-04-16T05:52:43.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.423+0000 7f911cc00700 1 -- 172.21.15.27:0/576539210 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f910801c630 con 0x7f91200fd710 2024-04-16T05:52:43.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.424+0000 7f9126e00700 1 -- 172.21.15.27:0/576539210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9120111840 con 0x7f91200fd710 2024-04-16T05:52:43.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.425+0000 7f9126e00700 1 -- 172.21.15.27:0/576539210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9120101a60 con 0x7f91200fd710 2024-04-16T05:52:43.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.426+0000 7f911cc00700 1 -- 172.21.15.27:0/576539210 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f910800bbf0 con 0x7f91200fd710 2024-04-16T05:52:43.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.427+0000 7f911cc00700 1 --2- 172.21.15.27:0/576539210 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f910406ec10 0x7f91040710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:43.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.427+0000 7f911cc00700 1 -- 172.21.15.27:0/576539210 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f910808fdc0 con 0x7f91200fd710 2024-04-16T05:52:43.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.427+0000 7f911fe00700 1 --2- 172.21.15.27:0/576539210 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f910406ec10 0x7f91040710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:43.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.428+0000 7f911fe00700 1 --2- 172.21.15.27:0/576539210 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f910406ec10 0x7f91040710c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f91200fdbc0 tx=0x7f9110009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:43.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.432+0000 7f911cc00700 1 -- 172.21.15.27:0/576539210 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9108093030 con 0x7f91200fd710 2024-04-16T05:52:43.525 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:43 smithi027 ceph-mon[93635]: from='client.34524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:52:43.525 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:43 smithi027 ceph-mon[93635]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:52:43.525 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:43 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/4080168671' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:43.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.728+0000 7f9126e00700 1 -- 172.21.15.27:0/576539210 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f9120111ab0 con 0x7f91200fd710 2024-04-16T05:52:43.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.729+0000 7f911cc00700 1 -- 172.21.15.27:0/576539210 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1936 (secure 0 0 0) 0x7f910805c500 con 0x7f91200fd710 2024-04-16T05:52:43.728 INFO:teuthology.orchestra.run.smithi027.stdout:e36 2024-04-16T05:52:43.728 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:52:43.728 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:52:43.728 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:52:43.728 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:52:43.729 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:52:43.729 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:52:43.729 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 35 2024-04-16T05:52:43.729 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:52:43.729 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:52:43.729 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:51:47.041625+0000 2024-04-16T05:52:43.729 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:52:43.729 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:52:43.729 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:52:43.729 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:52:43.729 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:52:43.729 INFO:teuthology.orchestra.run.smithi027.stdout:max_xattr_size 65536 2024-04-16T05:52:43.729 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:52:43.729 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:52:43.729 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 88 2024-04-16T05:52:43.730 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-16T05:52:43.730 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:52:43.730 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:52:43.730 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=34400} 2024-04-16T05:52:43.730 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:52:43.730 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:52:43.730 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:52:43.730 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:52:43.730 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:52:43.730 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:52:43.730 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:52:43.730 INFO:teuthology.orchestra.run.smithi027.stdout:bal_rank_mask -1 2024-04-16T05:52:43.730 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:52:43.730 INFO:teuthology.orchestra.run.smithi027.stdout:qdb_cluster leader: 34400 members: 34400 2024-04-16T05:52:43.731 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{0:34400} state up:active seq 23 join_fscid=1 addr [v2:172.21.15.27:6826/2831468256,v1:172.21.15.27:6827/2831468256] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:52:43.731 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:52:43.731 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:52:43.731 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:52:43.731 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:52:43.731 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{-1:34432} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6828/3219609959,v1:172.21.15.27:6829/3219609959] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:52:43.731 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:44337} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/97103507,v1:172.21.15.122:6825/97103507] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:52:43.731 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:44359} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/1030624816,v1:172.21.15.122:6827/1030624816] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:52:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.732+0000 7f9126e00700 1 -- 172.21.15.27:0/576539210 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f910406ec10 msgr2=0x7f91040710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.732+0000 7f9126e00700 1 --2- 172.21.15.27:0/576539210 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f910406ec10 0x7f91040710c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f91200fdbc0 tx=0x7f9110009040 comp rx=0 tx=0).stop 2024-04-16T05:52:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.732+0000 7f9126e00700 1 -- 172.21.15.27:0/576539210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91200fd710 msgr2=0x7f91201102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.732+0000 7f9126e00700 1 --2- 172.21.15.27:0/576539210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91200fd710 0x7f91201102c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f910800b970 tx=0x7f9108009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.732+0000 7f9126e00700 1 -- 172.21.15.27:0/576539210 shutdown_connections 2024-04-16T05:52:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.732+0000 7f9126e00700 1 --2- 172.21.15.27:0/576539210 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f910406ec10 0x7f91040710c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.732+0000 7f9126e00700 1 --2- 172.21.15.27:0/576539210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91200fcdc0 0x7f912010fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.732+0000 7f9126e00700 1 --2- 172.21.15.27:0/576539210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91200fd710 0x7f91201102c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:43.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.732+0000 7f9126e00700 1 -- 172.21.15.27:0/576539210 >> 172.21.15.27:0/576539210 conn(0x7f91200f8540 msgr2=0x7f91200f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.732+0000 7f9126e00700 1 -- 172.21.15.27:0/576539210 shutdown_connections 2024-04-16T05:52:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.732+0000 7f9126e00700 1 -- 172.21.15.27:0/576539210 wait complete. 2024-04-16T05:52:43.733 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 36 2024-04-16T05:52:43.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:43 smithi122 ceph-mon[80930]: from='client.34524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:52:43.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:43 smithi122 ceph-mon[80930]: from='client.34528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:52:43.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:43 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/4080168671' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:43.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.886+0000 7f6fc1800700 1 -- 172.21.15.27:0/1666551154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fbc103420 msgr2=0x7f6fbc103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:43.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.886+0000 7f6fc1800700 1 --2- 172.21.15.27:0/1666551154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fbc103420 0x7f6fbc103830 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6fac004940 tx=0x7f6fac009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:43.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.886+0000 7f6fc1800700 1 -- 172.21.15.27:0/1666551154 shutdown_connections 2024-04-16T05:52:43.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.886+0000 7f6fc1800700 1 --2- 172.21.15.27:0/1666551154 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fbc1046d0 0x7f6fbc104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:43.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.886+0000 7f6fc1800700 1 --2- 172.21.15.27:0/1666551154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fbc103420 0x7f6fbc103830 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:43.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.886+0000 7f6fc1800700 1 -- 172.21.15.27:0/1666551154 >> 172.21.15.27:0/1666551154 conn(0x7f6fbc072640 msgr2=0x7f6fbc074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:43.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.886+0000 7f6fc1800700 1 -- 172.21.15.27:0/1666551154 shutdown_connections 2024-04-16T05:52:43.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.886+0000 7f6fc1800700 1 -- 172.21.15.27:0/1666551154 wait complete. 2024-04-16T05:52:43.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.887+0000 7f6fc1800700 1 Processor -- start 2024-04-16T05:52:43.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.887+0000 7f6fc1800700 1 -- start start 2024-04-16T05:52:43.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.888+0000 7f6fc1800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fbc103420 0x7f6fbc113f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:43.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.888+0000 7f6fc1800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fbc1046d0 0x7f6fbc1144a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:43.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.888+0000 7f6fc1800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc114aa0 con 0x7f6fbc103420 2024-04-16T05:52:43.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.888+0000 7f6fc1800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc114c10 con 0x7f6fbc1046d0 2024-04-16T05:52:43.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.888+0000 7f6fbaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fbc103420 0x7f6fbc113f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:43.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.888+0000 7f6fbaa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fbc103420 0x7f6fbc113f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:36456/0 (socket says 172.21.15.27:36456) 2024-04-16T05:52:43.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.888+0000 7f6fbaa00700 1 -- 172.21.15.27:0/3019058080 learned_addr learned my addr 172.21.15.27:0/3019058080 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:52:43.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.888+0000 7f6fba000700 1 --2- 172.21.15.27:0/3019058080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fbc1046d0 0x7f6fbc1144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:43.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.888+0000 7f6fbaa00700 1 -- 172.21.15.27:0/3019058080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fbc1046d0 msgr2=0x7f6fbc1144a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:43.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.888+0000 7f6fbaa00700 1 --2- 172.21.15.27:0/3019058080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fbc1046d0 0x7f6fbc1144a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:43.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.888+0000 7f6fbaa00700 1 -- 172.21.15.27:0/3019058080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fac005040 con 0x7f6fbc103420 2024-04-16T05:52:43.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.889+0000 7f6fbaa00700 1 --2- 172.21.15.27:0/3019058080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fbc103420 0x7f6fbc113f60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6fac00b3b0 tx=0x7f6fac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:43.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.889+0000 7f6fb2a00700 1 -- 172.21.15.27:0/3019058080 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fac00e040 con 0x7f6fbc103420 2024-04-16T05:52:43.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.889+0000 7f6fb2a00700 1 -- 172.21.15.27:0/3019058080 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6fac004290 con 0x7f6fbc103420 2024-04-16T05:52:43.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.889+0000 7f6fb2a00700 1 -- 172.21.15.27:0/3019058080 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fac01c630 con 0x7f6fbc103420 2024-04-16T05:52:43.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.889+0000 7f6fc1800700 1 -- 172.21.15.27:0/3019058080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fbc119610 con 0x7f6fbc103420 2024-04-16T05:52:43.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.889+0000 7f6fc1800700 1 -- 172.21.15.27:0/3019058080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fbc119bd0 con 0x7f6fbc103420 2024-04-16T05:52:43.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.890+0000 7f6fc1800700 1 -- 172.21.15.27:0/3019058080 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fbc04c710 con 0x7f6fbc103420 2024-04-16T05:52:43.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.891+0000 7f6fb2a00700 1 -- 172.21.15.27:0/3019058080 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6fac00bbf0 con 0x7f6fbc103420 2024-04-16T05:52:43.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.892+0000 7f6fb2a00700 1 --2- 172.21.15.27:0/3019058080 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6fa006ec00 0x7f6fa00710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:43.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.893+0000 7f6fb2a00700 1 -- 172.21.15.27:0/3019058080 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f6fac08fe90 con 0x7f6fbc103420 2024-04-16T05:52:43.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.893+0000 7f6fba000700 1 --2- 172.21.15.27:0/3019058080 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6fa006ec00 0x7f6fa00710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:43.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.893+0000 7f6fba000700 1 --2- 172.21.15.27:0/3019058080 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6fa006ec00 0x7f6fa00710b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6fa80041d0 tx=0x7f6fa8009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:43.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:43.897+0000 7f6fb2a00700 1 -- 172.21.15.27:0/3019058080 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6fac05c5d0 con 0x7f6fbc103420 2024-04-16T05:52:44.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.155+0000 7f6fc1800700 1 -- 172.21.15.27:0/3019058080 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fbc108a60 con 0x7f6fa006ec00 2024-04-16T05:52:44.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.156+0000 7f6fb2a00700 1 -- 172.21.15.27:0/3019058080 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+438 (secure 0 0 0) 0x7f6fbc108a60 con 0x7f6fa006ec00 2024-04-16T05:52:44.155 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:52:44.155 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T05:52:44.155 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:52:44.155 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:52:44.156 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [ 2024-04-16T05:52:44.156 INFO:teuthology.orchestra.run.smithi027.stdout: "mds", 2024-04-16T05:52:44.156 INFO:teuthology.orchestra.run.smithi027.stdout: "mon", 2024-04-16T05:52:44.156 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-16T05:52:44.156 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr", 2024-04-16T05:52:44.156 INFO:teuthology.orchestra.run.smithi027.stdout: "osd" 2024-04-16T05:52:44.156 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-16T05:52:44.156 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "16/21 daemons upgraded", 2024-04-16T05:52:44.156 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Currently upgrading prometheus daemons", 2024-04-16T05:52:44.156 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:52:44.156 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:52:44.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.159+0000 7f6fc1800700 1 -- 172.21.15.27:0/3019058080 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6fa006ec00 msgr2=0x7f6fa00710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:44.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.159+0000 7f6fc1800700 1 --2- 172.21.15.27:0/3019058080 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6fa006ec00 0x7f6fa00710b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6fa80041d0 tx=0x7f6fa8009040 comp rx=0 tx=0).stop 2024-04-16T05:52:44.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.159+0000 7f6fc1800700 1 -- 172.21.15.27:0/3019058080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fbc103420 msgr2=0x7f6fbc113f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:44.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.159+0000 7f6fc1800700 1 --2- 172.21.15.27:0/3019058080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fbc103420 0x7f6fbc113f60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6fac00b3b0 tx=0x7f6fac009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:44.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.159+0000 7f6fc1800700 1 -- 172.21.15.27:0/3019058080 shutdown_connections 2024-04-16T05:52:44.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.159+0000 7f6fc1800700 1 --2- 172.21.15.27:0/3019058080 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6fa006ec00 0x7f6fa00710b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:44.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.159+0000 7f6fc1800700 1 --2- 172.21.15.27:0/3019058080 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fbc103420 0x7f6fbc113f60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:44.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.159+0000 7f6fc1800700 1 --2- 172.21.15.27:0/3019058080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fbc1046d0 0x7f6fbc1144a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:44.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.159+0000 7f6fc1800700 1 -- 172.21.15.27:0/3019058080 >> 172.21.15.27:0/3019058080 conn(0x7f6fbc072640 msgr2=0x7f6fbc073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:44.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.159+0000 7f6fc1800700 1 -- 172.21.15.27:0/3019058080 shutdown_connections 2024-04-16T05:52:44.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.160+0000 7f6fc1800700 1 -- 172.21.15.27:0/3019058080 wait complete. 2024-04-16T05:52:44.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.324+0000 7fa3e8a00700 1 -- 172.21.15.27:0/3717476305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e4066290 msgr2=0x7fa3e40ffa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:44.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.324+0000 7fa3e8a00700 1 --2- 172.21.15.27:0/3717476305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e4066290 0x7fa3e40ffa90 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4004940 tx=0x7fa3d4009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.324+0000 7fa3e8a00700 1 -- 172.21.15.27:0/3717476305 shutdown_connections 2024-04-16T05:52:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.324+0000 7fa3e8a00700 1 --2- 172.21.15.27:0/3717476305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3e40fffd0 0x7fa3e41023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.324+0000 7fa3e8a00700 1 --2- 172.21.15.27:0/3717476305 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e4066290 0x7fa3e40ffa90 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.324+0000 7fa3e8a00700 1 -- 172.21.15.27:0/3717476305 >> 172.21.15.27:0/3717476305 conn(0x7fa3e40f74c0 msgr2=0x7fa3e40f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.324+0000 7fa3e8a00700 1 -- 172.21.15.27:0/3717476305 shutdown_connections 2024-04-16T05:52:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.324+0000 7fa3e8a00700 1 -- 172.21.15.27:0/3717476305 wait complete. 2024-04-16T05:52:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.325+0000 7fa3e8a00700 1 Processor -- start 2024-04-16T05:52:44.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.326+0000 7fa3e8a00700 1 -- start start 2024-04-16T05:52:44.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.326+0000 7fa3e8a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e4066290 0x7fa3e410fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:44.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.326+0000 7fa3e8a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3e40fffd0 0x7fa3e4110260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:44.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.326+0000 7fa3e8a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e4110860 con 0x7fa3e4066290 2024-04-16T05:52:44.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.326+0000 7fa3e8a00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e4110fa0 con 0x7fa3e40fffd0 2024-04-16T05:52:44.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.326+0000 7fa3e1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e4066290 0x7fa3e410fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:44.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.326+0000 7fa3e1e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e4066290 0x7fa3e410fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46990/0 (socket says 172.21.15.27:46990) 2024-04-16T05:52:44.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.326+0000 7fa3e1e00700 1 -- 172.21.15.27:0/1890361285 learned_addr learned my addr 172.21.15.27:0/1890361285 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:52:44.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.327+0000 7fa3e1400700 1 --2- 172.21.15.27:0/1890361285 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3e40fffd0 0x7fa3e4110260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:44.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.327+0000 7fa3e1e00700 1 -- 172.21.15.27:0/1890361285 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3e40fffd0 msgr2=0x7fa3e4110260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:44.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.327+0000 7fa3e1e00700 1 --2- 172.21.15.27:0/1890361285 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3e40fffd0 0x7fa3e4110260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:44.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.327+0000 7fa3e1e00700 1 -- 172.21.15.27:0/1890361285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d4005040 con 0x7fa3e4066290 2024-04-16T05:52:44.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.327+0000 7fa3e1400700 1 --2- 172.21.15.27:0/1890361285 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3e40fffd0 0x7fa3e4110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:52:44.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.327+0000 7fa3e1e00700 1 --2- 172.21.15.27:0/1890361285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e4066290 0x7fa3e410fd20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa3d400b970 tx=0x7fa3d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:44.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.328+0000 7fa3da000700 1 -- 172.21.15.27:0/1890361285 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3d400e040 con 0x7fa3e4066290 2024-04-16T05:52:44.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.328+0000 7fa3da000700 1 -- 172.21.15.27:0/1890361285 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3d400bbf0 con 0x7fa3e4066290 2024-04-16T05:52:44.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.328+0000 7fa3e8a00700 1 -- 172.21.15.27:0/1890361285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e4111220 con 0x7fa3e4066290 2024-04-16T05:52:44.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.328+0000 7fa3da000700 1 -- 172.21.15.27:0/1890361285 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3d401c630 con 0x7fa3e4066290 2024-04-16T05:52:44.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.328+0000 7fa3e8a00700 1 -- 172.21.15.27:0/1890361285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e41117e0 con 0x7fa3e4066290 2024-04-16T05:52:44.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.329+0000 7fa3e8a00700 1 -- 172.21.15.27:0/1890361285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e405f5f0 con 0x7fa3e4066290 2024-04-16T05:52:44.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.330+0000 7fa3da000700 1 -- 172.21.15.27:0/1890361285 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa3d400bd60 con 0x7fa3e4066290 2024-04-16T05:52:44.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.332+0000 7fa3da000700 1 --2- 172.21.15.27:0/1890361285 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa3c806ec10 0x7fa3c80710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:52:44.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.332+0000 7fa3da000700 1 -- 172.21.15.27:0/1890361285 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7fa3d408fe20 con 0x7fa3e4066290 2024-04-16T05:52:44.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.332+0000 7fa3e1400700 1 --2- 172.21.15.27:0/1890361285 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa3c806ec10 0x7fa3c80710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:52:44.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.333+0000 7fa3e1400700 1 --2- 172.21.15.27:0/1890361285 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa3c806ec10 0x7fa3c80710c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0007fd0 tx=0x7fa3d0009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:52:44.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.337+0000 7fa3da000700 1 -- 172.21.15.27:0/1890361285 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa3d405c560 con 0x7fa3e4066290 2024-04-16T05:52:44.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.689+0000 7fa3e8a00700 1 -- 172.21.15.27:0/1890361285 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fa3e4111bb0 con 0x7fa3e4066290 2024-04-16T05:52:44.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.690+0000 7fa3da000700 1 -- 172.21.15.27:0/1890361285 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fa3d405c0f0 con 0x7fa3e4066290 2024-04-16T05:52:44.689 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:52:44.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.693+0000 7fa3e8a00700 1 -- 172.21.15.27:0/1890361285 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa3c806ec10 msgr2=0x7fa3c80710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:44.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.693+0000 7fa3e8a00700 1 --2- 172.21.15.27:0/1890361285 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa3c806ec10 0x7fa3c80710c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0007fd0 tx=0x7fa3d0009040 comp rx=0 tx=0).stop 2024-04-16T05:52:44.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.693+0000 7fa3e8a00700 1 -- 172.21.15.27:0/1890361285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e4066290 msgr2=0x7fa3e410fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:52:44.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.693+0000 7fa3e8a00700 1 --2- 172.21.15.27:0/1890361285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e4066290 0x7fa3e410fd20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa3d400b970 tx=0x7fa3d4009e30 comp rx=0 tx=0).stop 2024-04-16T05:52:44.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.693+0000 7fa3e8a00700 1 -- 172.21.15.27:0/1890361285 shutdown_connections 2024-04-16T05:52:44.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.693+0000 7fa3e8a00700 1 --2- 172.21.15.27:0/1890361285 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa3c806ec10 0x7fa3c80710c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:44.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.693+0000 7fa3e8a00700 1 --2- 172.21.15.27:0/1890361285 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa3e4066290 0x7fa3e410fd20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:44.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.693+0000 7fa3e8a00700 1 --2- 172.21.15.27:0/1890361285 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3e40fffd0 0x7fa3e4110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:52:44.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.693+0000 7fa3e8a00700 1 -- 172.21.15.27:0/1890361285 >> 172.21.15.27:0/1890361285 conn(0x7fa3e40f74c0 msgr2=0x7fa3e40fbad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:52:44.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.693+0000 7fa3e8a00700 1 -- 172.21.15.27:0/1890361285 shutdown_connections 2024-04-16T05:52:44.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:52:44.693+0000 7fa3e8a00700 1 -- 172.21.15.27:0/1890361285 wait complete. 2024-04-16T05:52:44.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:44 smithi122 ceph-mon[80930]: from='client.34532 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:52:44.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:44 smithi122 ceph-mon[80930]: pgmap v259: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:44.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:44 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/576539210' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:52:44.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:44 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:52:44.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:44 smithi027 ceph-mon[93635]: from='client.34532 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:52:44.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:44 smithi027 ceph-mon[93635]: pgmap v259: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:44.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:44 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/576539210' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:52:44.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:52:45.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:45 smithi122 ceph-mon[80930]: from='client.34544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:52:45.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:45 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1890361285' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:52:45.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:45 smithi027 ceph-mon[93635]: from='client.34544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:52:45.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:45 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1890361285' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:52:46.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:46 smithi122 ceph-mon[80930]: pgmap v260: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:46.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:46 smithi027 ceph-mon[93635]: pgmap v260: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:48.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:48 smithi027 ceph-mon[93635]: pgmap v261: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:49.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:48 smithi122 ceph-mon[80930]: pgmap v261: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:50.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:49 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:50.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:49 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:50.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:49 smithi122 ceph-mon[80930]: pgmap v262: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:50.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:49 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:50.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:49 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:50.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:49 smithi027 ceph-mon[93635]: pgmap v262: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:52.499 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:52 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:52.500 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:52 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:52.500 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:52 smithi027 ceph-mon[93635]: pgmap v263: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:52.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:52 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:52.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:52 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:52.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:52 smithi122 ceph-mon[80930]: pgmap v263: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:54.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:54 smithi027 ceph-mon[93635]: pgmap v264: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:54.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:54 smithi122 ceph-mon[80930]: pgmap v264: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:55.596 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 534, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134076 2024-04-16T05:52:55.597 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 534, success, outsize: 120 2024-04-16T05:52:55.621 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 536, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134173 2024-04-16T05:52:55.621 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 536, success, outsize: 120 2024-04-16T05:52:55.652 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 538, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134273 2024-04-16T05:52:55.652 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 538, success, outsize: 120 2024-04-16T05:52:55.685 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 540, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134379 2024-04-16T05:52:55.686 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 540, success, outsize: 120 2024-04-16T05:52:55.717 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 542, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134482 2024-04-16T05:52:55.718 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 542, success, outsize: 120 2024-04-16T05:52:55.750 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 544, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134579 2024-04-16T05:52:55.750 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 544, success, outsize: 120 2024-04-16T05:52:55.781 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 546, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134679 2024-04-16T05:52:55.781 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 546, success, outsize: 120 2024-04-16T05:52:55.815 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 548, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134785 2024-04-16T05:52:55.816 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 548, success, outsize: 120 2024-04-16T05:52:55.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 550, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134888 2024-04-16T05:52:55.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 550, success, outsize: 120 2024-04-16T05:52:55.873 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 552, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 134985 2024-04-16T05:52:55.873 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 552, success, outsize: 120 2024-04-16T05:52:55.904 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 554, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 135085 2024-04-16T05:52:55.905 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 554, success, outsize: 120 2024-04-16T05:52:55.938 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 556, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 135191 2024-04-16T05:52:55.938 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 556, success, outsize: 120 2024-04-16T05:52:55.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 558, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 135294 2024-04-16T05:52:55.969 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 558, success, outsize: 120 2024-04-16T05:52:55.999 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 560, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 135391 2024-04-16T05:52:56.000 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 560, success, outsize: 120 2024-04-16T05:52:56.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 562, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 135491 2024-04-16T05:52:56.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 562, success, outsize: 120 2024-04-16T05:52:56.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 564, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 135597 2024-04-16T05:52:56.055 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 564, success, outsize: 120 2024-04-16T05:52:56.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 566, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 135700 2024-04-16T05:52:56.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 566, success, outsize: 120 2024-04-16T05:52:56.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 568, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 135797 2024-04-16T05:52:56.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 568, success, outsize: 120 2024-04-16T05:52:56.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 570, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 135897 2024-04-16T05:52:56.146 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 570, success, outsize: 120 2024-04-16T05:52:56.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 572, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 136003 2024-04-16T05:52:56.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 572, success, outsize: 120 2024-04-16T05:52:56.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 574, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 136106 2024-04-16T05:52:56.201 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 574, success, outsize: 120 2024-04-16T05:52:56.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 576, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 136203 2024-04-16T05:52:56.227 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 576, success, outsize: 120 2024-04-16T05:52:56.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 578, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 136303 2024-04-16T05:52:56.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 578, success, outsize: 120 2024-04-16T05:52:56.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 580, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 136409 2024-04-16T05:52:56.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 580, success, outsize: 120 2024-04-16T05:52:56.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 582, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 136512 2024-04-16T05:52:56.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 582, success, outsize: 120 2024-04-16T05:52:56.329 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 584, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 136609 2024-04-16T05:52:56.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 584, success, outsize: 120 2024-04-16T05:52:56.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 586, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 136709 2024-04-16T05:52:56.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 586, success, outsize: 120 2024-04-16T05:52:56.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 588, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 136815 2024-04-16T05:52:56.390 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 588, success, outsize: 120 2024-04-16T05:52:56.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:56 smithi122 ceph-mon[80930]: pgmap v265: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:56.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:56 smithi027 ceph-mon[93635]: pgmap v265: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:58.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:58.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:58.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:52:58.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:52:58.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:58.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: pgmap v266: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:58.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T05:52:58.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T05:52:58.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:52:58.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.523 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.524 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:58 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:58.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:58.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:52:58.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:52:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:52:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: pgmap v266: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:52:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T05:52:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-16T05:52:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:52:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:58.558 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:58 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:52:59.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:52:59 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:52:59.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:52:59 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:53:00.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:00 smithi122 ceph-mon[80930]: pgmap v267: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:00.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:00 smithi027 ceph-mon[93635]: pgmap v267: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:01.394 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:01 smithi027 ceph-mon[93635]: Upgrade: Updating alertmanager.smithi027 2024-04-16T05:53:01.394 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:01 smithi027 ceph-mon[93635]: Deploying daemon alertmanager.smithi027 on smithi027 2024-04-16T05:53:01.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:01 smithi122 ceph-mon[80930]: Upgrade: Updating alertmanager.smithi027 2024-04-16T05:53:01.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:01 smithi122 ceph-mon[80930]: Deploying daemon alertmanager.smithi027 on smithi027 2024-04-16T05:53:02.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:02 smithi027 ceph-mon[93635]: pgmap v268: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:02.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:02 smithi122 ceph-mon[80930]: pgmap v268: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:04.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:04 smithi122 ceph-mon[80930]: pgmap v269: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:04.779 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:04 smithi027 ceph-mon[93635]: pgmap v269: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:06.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:06 smithi122 ceph-mon[80930]: pgmap v270: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:06.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:06 smithi027 ceph-mon[93635]: pgmap v270: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:08.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:08 smithi122 ceph-mon[80930]: pgmap v271: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:08.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:08 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:08.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:08 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:08.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:08 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:53:08.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:08 smithi027 ceph-mon[93635]: pgmap v271: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:08.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:08 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:08.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:08 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:08.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:08 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:53:10.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:10 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:10.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:10 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:10.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:10 smithi122 ceph-mon[80930]: pgmap v272: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:10.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:10 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:10.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:10 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:10.307 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:10 smithi027 ceph-mon[93635]: pgmap v272: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:12.709 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:12 smithi027 ceph-mon[93635]: pgmap v273: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:12.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:12 smithi122 ceph-mon[80930]: pgmap v273: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:14.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:14 smithi122 ceph-mon[80930]: pgmap v274: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:14.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:53:14.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:14 smithi027 ceph-mon[93635]: pgmap v274: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:14.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:53:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.870+0000 7f6ec0a00700 1 -- 172.21.15.27:0/4190941519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ebc0fff70 msgr2=0x7f6ebc102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.870+0000 7f6ec0a00700 1 --2- 172.21.15.27:0/4190941519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ebc0fff70 0x7f6ebc102390 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4004940 tx=0x7f6ea4009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.870+0000 7f6ec0a00700 1 -- 172.21.15.27:0/4190941519 shutdown_connections 2024-04-16T05:53:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.870+0000 7f6ec0a00700 1 --2- 172.21.15.27:0/4190941519 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ebc0fff70 0x7f6ebc102390 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:14.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.870+0000 7f6ec0a00700 1 --2- 172.21.15.27:0/4190941519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ebc0662f0 0x7f6ebc0ffa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:14.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.871+0000 7f6ec0a00700 1 -- 172.21.15.27:0/4190941519 >> 172.21.15.27:0/4190941519 conn(0x7f6ebc0f74c0 msgr2=0x7f6ebc0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:14.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.871+0000 7f6ec0a00700 1 -- 172.21.15.27:0/4190941519 shutdown_connections 2024-04-16T05:53:14.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.871+0000 7f6ec0a00700 1 -- 172.21.15.27:0/4190941519 wait complete. 2024-04-16T05:53:14.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.871+0000 7f6ec0a00700 1 Processor -- start 2024-04-16T05:53:14.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.872+0000 7f6ec0a00700 1 -- start start 2024-04-16T05:53:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.872+0000 7f6ec0a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ebc0662f0 0x7f6ebc10fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.872+0000 7f6ec0a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ebc0fff70 0x7f6ebc110280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.872+0000 7f6ec0a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc110880 con 0x7f6ebc0662f0 2024-04-16T05:53:14.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.872+0000 7f6ec0a00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc110fc0 con 0x7f6ebc0fff70 2024-04-16T05:53:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.872+0000 7f6eb9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ebc0662f0 0x7f6ebc10fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.872+0000 7f6eb9e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ebc0662f0 0x7f6ebc10fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54538/0 (socket says 172.21.15.27:54538) 2024-04-16T05:53:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.872+0000 7f6eb9e00700 1 -- 172.21.15.27:0/2086177730 learned_addr learned my addr 172.21.15.27:0/2086177730 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:53:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.873+0000 7f6eb9e00700 1 -- 172.21.15.27:0/2086177730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ebc0fff70 msgr2=0x7f6ebc110280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.873+0000 7f6eb3e00700 1 --2- 172.21.15.27:0/2086177730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ebc0fff70 0x7f6ebc110280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.873+0000 7f6eb9e00700 1 --2- 172.21.15.27:0/2086177730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ebc0fff70 0x7f6ebc110280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.873+0000 7f6eb9e00700 1 -- 172.21.15.27:0/2086177730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea4005040 con 0x7f6ebc0662f0 2024-04-16T05:53:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.873+0000 7f6eb9e00700 1 --2- 172.21.15.27:0/2086177730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ebc0662f0 0x7f6ebc10fd40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6eac00c800 tx=0x7f6eac004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.873+0000 7f6eb2000700 1 -- 172.21.15.27:0/2086177730 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6eac00a760 con 0x7f6ebc0662f0 2024-04-16T05:53:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.873+0000 7f6eb2000700 1 -- 172.21.15.27:0/2086177730 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6eac00ad90 con 0x7f6ebc0662f0 2024-04-16T05:53:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.873+0000 7f6ec0a00700 1 -- 172.21.15.27:0/2086177730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ebc1112a0 con 0x7f6ebc0662f0 2024-04-16T05:53:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.873+0000 7f6eb2000700 1 -- 172.21.15.27:0/2086177730 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6eac0113a0 con 0x7f6ebc0662f0 2024-04-16T05:53:14.872 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.874+0000 7f6ec0a00700 1 -- 172.21.15.27:0/2086177730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ebc111860 con 0x7f6ebc0662f0 2024-04-16T05:53:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.875+0000 7f6ec0a00700 1 -- 172.21.15.27:0/2086177730 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ebc0f8ff0 con 0x7f6ebc0662f0 2024-04-16T05:53:14.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.876+0000 7f6eb2000700 1 -- 172.21.15.27:0/2086177730 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6eac012400 con 0x7f6ebc0662f0 2024-04-16T05:53:14.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.877+0000 7f6eb2000700 1 --2- 172.21.15.27:0/2086177730 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6ea006ec10 0x7f6ea00710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:14.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.877+0000 7f6eb2000700 1 -- 172.21.15.27:0/2086177730 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f6eac08eaa0 con 0x7f6ebc0662f0 2024-04-16T05:53:14.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.877+0000 7f6eb3e00700 1 --2- 172.21.15.27:0/2086177730 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6ea006ec10 0x7f6ea00710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:14.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.878+0000 7f6eb3e00700 1 --2- 172.21.15.27:0/2086177730 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6ea006ec10 0x7f6ea00710c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6ea400b970 tx=0x7f6ea4009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:14.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:14.882+0000 7f6eb2000700 1 -- 172.21.15.27:0/2086177730 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6eac05b200 con 0x7f6ebc0662f0 2024-04-16T05:53:15.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.158+0000 7f6ec0a00700 1 -- 172.21.15.27:0/2086177730 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ebc111ae0 con 0x7f6ea006ec10 2024-04-16T05:53:15.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.160+0000 7f6eb2000700 1 -- 172.21.15.27:0/2086177730 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+440 (secure 0 0 0) 0x7f6ebc111ae0 con 0x7f6ea006ec10 2024-04-16T05:53:15.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.163+0000 7f6ec0a00700 1 -- 172.21.15.27:0/2086177730 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6ea006ec10 msgr2=0x7f6ea00710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:15.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.163+0000 7f6ec0a00700 1 --2- 172.21.15.27:0/2086177730 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6ea006ec10 0x7f6ea00710c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6ea400b970 tx=0x7f6ea4009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:15.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.163+0000 7f6ec0a00700 1 -- 172.21.15.27:0/2086177730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ebc0662f0 msgr2=0x7f6ebc10fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:15.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.163+0000 7f6ec0a00700 1 --2- 172.21.15.27:0/2086177730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ebc0662f0 0x7f6ebc10fd40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6eac00c800 tx=0x7f6eac004470 comp rx=0 tx=0).stop 2024-04-16T05:53:15.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.163+0000 7f6ec0a00700 1 -- 172.21.15.27:0/2086177730 shutdown_connections 2024-04-16T05:53:15.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.163+0000 7f6ec0a00700 1 --2- 172.21.15.27:0/2086177730 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6ea006ec10 0x7f6ea00710c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:15.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.163+0000 7f6ec0a00700 1 --2- 172.21.15.27:0/2086177730 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ebc0662f0 0x7f6ebc10fd40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:15.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.163+0000 7f6ec0a00700 1 --2- 172.21.15.27:0/2086177730 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ebc0fff70 0x7f6ebc110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:15.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.163+0000 7f6ec0a00700 1 -- 172.21.15.27:0/2086177730 >> 172.21.15.27:0/2086177730 conn(0x7f6ebc0f74c0 msgr2=0x7f6ebc0fbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:15.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.163+0000 7f6ec0a00700 1 -- 172.21.15.27:0/2086177730 shutdown_connections 2024-04-16T05:53:15.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.164+0000 7f6ec0a00700 1 -- 172.21.15.27:0/2086177730 wait complete. 2024-04-16T05:53:15.178 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:53:15.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.328+0000 7f2c7ec00700 1 -- 172.21.15.27:0/1915891394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c780fcfa0 msgr2=0x7f2c780fd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:15.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.328+0000 7f2c7ec00700 1 --2- 172.21.15.27:0/1915891394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c780fcfa0 0x7f2c780fd410 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2c64004940 tx=0x7f2c64009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:15.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.328+0000 7f2c7ec00700 1 -- 172.21.15.27:0/1915891394 shutdown_connections 2024-04-16T05:53:15.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.328+0000 7f2c7ec00700 1 --2- 172.21.15.27:0/1915891394 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c780fcfa0 0x7f2c780fd410 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:15.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.328+0000 7f2c7ec00700 1 --2- 172.21.15.27:0/1915891394 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c780fbcf0 0x7f2c780fc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:15.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.328+0000 7f2c7ec00700 1 -- 172.21.15.27:0/1915891394 >> 172.21.15.27:0/1915891394 conn(0x7f2c780f7450 msgr2=0x7f2c780f9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:15.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.328+0000 7f2c7ec00700 1 -- 172.21.15.27:0/1915891394 shutdown_connections 2024-04-16T05:53:15.327 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.329+0000 7f2c7ec00700 1 -- 172.21.15.27:0/1915891394 wait complete. 2024-04-16T05:53:15.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.329+0000 7f2c7ec00700 1 Processor -- start 2024-04-16T05:53:15.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.330+0000 7f2c7ec00700 1 -- start start 2024-04-16T05:53:15.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.330+0000 7f2c7ec00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c780fbcf0 0x7f2c7810fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:15.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.330+0000 7f2c7ec00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c780fcfa0 0x7f2c78110170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:15.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.330+0000 7f2c7ec00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c78110770 con 0x7f2c780fcfa0 2024-04-16T05:53:15.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.330+0000 7f2c7ec00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c78110f50 con 0x7f2c780fbcf0 2024-04-16T05:53:15.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.330+0000 7f2c77400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c780fcfa0 0x7f2c78110170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:15.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.331+0000 7f2c77400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c780fcfa0 0x7f2c78110170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54562/0 (socket says 172.21.15.27:54562) 2024-04-16T05:53:15.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.331+0000 7f2c77400700 1 -- 172.21.15.27:0/3838080236 learned_addr learned my addr 172.21.15.27:0/3838080236 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:53:15.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.331+0000 7f2c77e00700 1 --2- 172.21.15.27:0/3838080236 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c780fbcf0 0x7f2c7810fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:15.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.331+0000 7f2c77400700 1 -- 172.21.15.27:0/3838080236 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c780fbcf0 msgr2=0x7f2c7810fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:15.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.331+0000 7f2c77400700 1 --2- 172.21.15.27:0/3838080236 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c780fbcf0 0x7f2c7810fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:15.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.331+0000 7f2c77400700 1 -- 172.21.15.27:0/3838080236 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c64005040 con 0x7f2c780fcfa0 2024-04-16T05:53:15.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.332+0000 7f2c77400700 1 --2- 172.21.15.27:0/3838080236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c780fcfa0 0x7f2c78110170 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2c64009150 tx=0x7f2c64009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:15.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.332+0000 7f2c6fe00700 1 -- 172.21.15.27:0/3838080236 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c6400e040 con 0x7f2c780fcfa0 2024-04-16T05:53:15.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.332+0000 7f2c6fe00700 1 -- 172.21.15.27:0/3838080236 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c6400bbf0 con 0x7f2c780fcfa0 2024-04-16T05:53:15.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.332+0000 7f2c7ec00700 1 -- 172.21.15.27:0/3838080236 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c781111d0 con 0x7f2c780fcfa0 2024-04-16T05:53:15.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.332+0000 7f2c6fe00700 1 -- 172.21.15.27:0/3838080236 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2c6401c630 con 0x7f2c780fcfa0 2024-04-16T05:53:15.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.332+0000 7f2c7ec00700 1 -- 172.21.15.27:0/3838080236 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c78111710 con 0x7f2c780fcfa0 2024-04-16T05:53:15.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.333+0000 7f2c7ec00700 1 -- 172.21.15.27:0/3838080236 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c78101930 con 0x7f2c780fcfa0 2024-04-16T05:53:15.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.335+0000 7f2c6fe00700 1 -- 172.21.15.27:0/3838080236 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2c6400bd60 con 0x7f2c780fcfa0 2024-04-16T05:53:15.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.336+0000 7f2c6fe00700 1 --2- 172.21.15.27:0/3838080236 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2c5c06ec10 0x7f2c5c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:15.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.336+0000 7f2c6fe00700 1 -- 172.21.15.27:0/3838080236 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f2c6408fe30 con 0x7f2c780fcfa0 2024-04-16T05:53:15.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.337+0000 7f2c77e00700 1 --2- 172.21.15.27:0/3838080236 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2c5c06ec10 0x7f2c5c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:15.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.337+0000 7f2c77e00700 1 --2- 172.21.15.27:0/3838080236 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2c5c06ec10 0x7f2c5c0710c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2c68007fd0 tx=0x7f2c68009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:15.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.341+0000 7f2c6fe00700 1 -- 172.21.15.27:0/3838080236 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2c6405c570 con 0x7f2c780fcfa0 2024-04-16T05:53:15.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.609+0000 7f2c7ec00700 1 -- 172.21.15.27:0/3838080236 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c78111a40 con 0x7f2c5c06ec10 2024-04-16T05:53:15.609 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.611+0000 7f2c6fe00700 1 -- 172.21.15.27:0/3838080236 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+440 (secure 0 0 0) 0x7f2c78111a40 con 0x7f2c5c06ec10 2024-04-16T05:53:15.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.614+0000 7f2c7ec00700 1 -- 172.21.15.27:0/3838080236 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2c5c06ec10 msgr2=0x7f2c5c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:15.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.614+0000 7f2c7ec00700 1 --2- 172.21.15.27:0/3838080236 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2c5c06ec10 0x7f2c5c0710c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2c68007fd0 tx=0x7f2c68009040 comp rx=0 tx=0).stop 2024-04-16T05:53:15.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.614+0000 7f2c7ec00700 1 -- 172.21.15.27:0/3838080236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c780fcfa0 msgr2=0x7f2c78110170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:15.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.614+0000 7f2c7ec00700 1 --2- 172.21.15.27:0/3838080236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c780fcfa0 0x7f2c78110170 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2c64009150 tx=0x7f2c64009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.614+0000 7f2c7ec00700 1 -- 172.21.15.27:0/3838080236 shutdown_connections 2024-04-16T05:53:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.614+0000 7f2c7ec00700 1 --2- 172.21.15.27:0/3838080236 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2c5c06ec10 0x7f2c5c0710c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.614+0000 7f2c7ec00700 1 --2- 172.21.15.27:0/3838080236 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2c780fbcf0 0x7f2c7810fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.614+0000 7f2c7ec00700 1 --2- 172.21.15.27:0/3838080236 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2c780fcfa0 0x7f2c78110170 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.614+0000 7f2c7ec00700 1 -- 172.21.15.27:0/3838080236 >> 172.21.15.27:0/3838080236 conn(0x7f2c780f7450 msgr2=0x7f2c78066410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.615+0000 7f2c7ec00700 1 -- 172.21.15.27:0/3838080236 shutdown_connections 2024-04-16T05:53:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.615+0000 7f2c7ec00700 1 -- 172.21.15.27:0/3838080236 wait complete. 2024-04-16T05:53:15.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.779+0000 7f53af200700 1 -- 172.21.15.27:0/1546990135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a80662f0 msgr2=0x7f53a80fd820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:15.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.779+0000 7f53af200700 1 --2- 172.21.15.27:0/1546990135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a80662f0 0x7f53a80fd820 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5398004940 tx=0x7f5398009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:15.777 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.779+0000 7f53af200700 1 -- 172.21.15.27:0/1546990135 shutdown_connections 2024-04-16T05:53:15.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.779+0000 7f53af200700 1 --2- 172.21.15.27:0/1546990135 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53a80fdd60 0x7f53a81001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:15.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.779+0000 7f53af200700 1 --2- 172.21.15.27:0/1546990135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a80662f0 0x7f53a80fd820 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:15.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.779+0000 7f53af200700 1 -- 172.21.15.27:0/1546990135 >> 172.21.15.27:0/1546990135 conn(0x7f53a80f74c0 msgr2=0x7f53a80f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:15.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.779+0000 7f53af200700 1 -- 172.21.15.27:0/1546990135 shutdown_connections 2024-04-16T05:53:15.778 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.780+0000 7f53af200700 1 -- 172.21.15.27:0/1546990135 wait complete. 2024-04-16T05:53:15.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.780+0000 7f53af200700 1 Processor -- start 2024-04-16T05:53:15.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.781+0000 7f53af200700 1 -- start start 2024-04-16T05:53:15.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.781+0000 7f53af200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a80662f0 0x7f53a8114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:15.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.781+0000 7f53af200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53a80fdd60 0x7f53a8114580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:15.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.781+0000 7f53af200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a8114b80 con 0x7f53a80662f0 2024-04-16T05:53:15.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.781+0000 7f53af200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a81153d0 con 0x7f53a80fdd60 2024-04-16T05:53:15.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.781+0000 7f53a7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a80662f0 0x7f53a8114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:15.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.781+0000 7f53a7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a80662f0 0x7f53a8114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54586/0 (socket says 172.21.15.27:54586) 2024-04-16T05:53:15.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.781+0000 7f53a7e00700 1 -- 172.21.15.27:0/1741047543 learned_addr learned my addr 172.21.15.27:0/1741047543 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:53:15.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.782+0000 7f53a7e00700 1 -- 172.21.15.27:0/1741047543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53a80fdd60 msgr2=0x7f53a8114580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:15.780 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.782+0000 7f53a7e00700 1 --2- 172.21.15.27:0/1741047543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53a80fdd60 0x7f53a8114580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:15.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.782+0000 7f53a7e00700 1 -- 172.21.15.27:0/1741047543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5398005040 con 0x7f53a80662f0 2024-04-16T05:53:15.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.782+0000 7f53a7e00700 1 --2- 172.21.15.27:0/1741047543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a80662f0 0x7f53a8114040 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f539800b970 tx=0x7f5398009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:15.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.782+0000 7f539fe00700 1 -- 172.21.15.27:0/1741047543 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f539800e040 con 0x7f53a80662f0 2024-04-16T05:53:15.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.782+0000 7f539fe00700 1 -- 172.21.15.27:0/1741047543 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5398004290 con 0x7f53a80662f0 2024-04-16T05:53:15.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.782+0000 7f539fe00700 1 -- 172.21.15.27:0/1741047543 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f539801c630 con 0x7f53a80662f0 2024-04-16T05:53:15.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.782+0000 7f53af200700 1 -- 172.21.15.27:0/1741047543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a8115650 con 0x7f53a80662f0 2024-04-16T05:53:15.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.783+0000 7f53af200700 1 -- 172.21.15.27:0/1741047543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a8115b40 con 0x7f53a80662f0 2024-04-16T05:53:15.781 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.784+0000 7f53af200700 1 -- 172.21.15.27:0/1741047543 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a805f5f0 con 0x7f53a80662f0 2024-04-16T05:53:15.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.785+0000 7f539fe00700 1 -- 172.21.15.27:0/1741047543 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f539800bbf0 con 0x7f53a80662f0 2024-04-16T05:53:15.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.787+0000 7f539fe00700 1 --2- 172.21.15.27:0/1741047543 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f538c06ec10 0x7f538c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:15.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.787+0000 7f539fe00700 1 -- 172.21.15.27:0/1741047543 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f53980908f0 con 0x7f53a80662f0 2024-04-16T05:53:15.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.788+0000 7f53a7400700 1 --2- 172.21.15.27:0/1741047543 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f538c06ec10 0x7f538c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:15.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.788+0000 7f53a7400700 1 --2- 172.21.15.27:0/1741047543 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f538c06ec10 0x7f538c0710c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5394004ad0 tx=0x7f5394007d40 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:15.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:15.791+0000 7f539fe00700 1 -- 172.21.15.27:0/1741047543 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f539805d030 con 0x7f53a80662f0 2024-04-16T05:53:16.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.043+0000 7f53af200700 1 -- 172.21.15.27:0/1741047543 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f53a8115fa0 con 0x7f538c06ec10 2024-04-16T05:53:16.060 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.062+0000 7f539fe00700 1 -- 172.21.15.27:0/1741047543 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f53a8115fa0 con 0x7f538c06ec10 2024-04-16T05:53:16.060 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:53:16.060 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (8s) 0s ago 20m 13.7M - 0.25.0 c8568f914cd2 42321467ded6 2024-04-16T05:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (6m) 0s ago 20m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 4aa1f0006685 2024-04-16T05:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (6m) 56s ago 19m 6656k - 19.0.0-2884-g218a6914 a4e4811815e4 1c1912507d56 2024-04-16T05:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (9m) 0s ago 19m 75.0M - 9.4.7 954c08fa6188 f7ee2a1afd12 2024-04-16T05:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (2m) 0s ago 12m 11.8M - 19.0.0-2884-g218a6914 a4e4811815e4 b3c2178c537e 2024-04-16T05:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (2m) 0s ago 12m 19.4M - 19.0.0-2884-g218a6914 a4e4811815e4 14b0c83f2891 2024-04-16T05:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (110s) 56s ago 12m 11.2M - 19.0.0-2884-g218a6914 a4e4811815e4 8ba95adfaaef 2024-04-16T05:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (85s) 56s ago 12m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 c60e9ca0f1c8 2024-04-16T05:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:8443,9283,8765 running (9m) 0s ago 22m 514M - 19.0.0-2884-g218a6914 a4e4811815e4 b69a0b08e12b 2024-04-16T05:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283,8765 running (8m) 56s ago 18m 454M - 19.0.0-2884-g218a6914 a4e4811815e4 32e27bcd7e42 2024-04-16T05:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (7m) 0s ago 22m 54.5M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 299ca337e4ca 2024-04-16T05:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (7m) 56s ago 18m 40.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5db5f3037ed0 2024-04-16T05:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (67s) 0s ago 20m 7944k - 1.5.0 0da6a335fe13 f3382c6beacb 2024-04-16T05:53:16.061 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (60s) 56s ago 19m 3078k - 1.5.0 0da6a335fe13 1f6817821f3a 2024-04-16T05:53:16.062 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (5m) 0s ago 17m 47.2M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 cef03b67bc5a 2024-04-16T05:53:16.062 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (5m) 0s ago 17m 46.1M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3a6ab1fc6e79 2024-04-16T05:53:16.062 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (4m) 0s ago 16m 44.7M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3524731d7277 2024-04-16T05:53:16.062 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (4m) 56s ago 15m 44.7M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 897d3579f929 2024-04-16T05:53:16.062 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (3m) 56s ago 15m 42.2M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 fa97758ff7c5 2024-04-16T05:53:16.062 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (3m) 56s ago 14m 49.6M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 fd12260a820d 2024-04-16T05:53:16.062 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (34s) 0s ago 19m 47.6M - 2.43.0 a07b618ecd1d a6cddf5443d5 2024-04-16T05:53:16.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.066+0000 7f53af200700 1 -- 172.21.15.27:0/1741047543 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f538c06ec10 msgr2=0x7f538c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:16.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.066+0000 7f53af200700 1 --2- 172.21.15.27:0/1741047543 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f538c06ec10 0x7f538c0710c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5394004ad0 tx=0x7f5394007d40 comp rx=0 tx=0).stop 2024-04-16T05:53:16.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.066+0000 7f53af200700 1 -- 172.21.15.27:0/1741047543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a80662f0 msgr2=0x7f53a8114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:16.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.066+0000 7f53af200700 1 --2- 172.21.15.27:0/1741047543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a80662f0 0x7f53a8114040 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f539800b970 tx=0x7f5398009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:16.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.066+0000 7f53af200700 1 -- 172.21.15.27:0/1741047543 shutdown_connections 2024-04-16T05:53:16.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.066+0000 7f53af200700 1 --2- 172.21.15.27:0/1741047543 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f538c06ec10 0x7f538c0710c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:16.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.066+0000 7f53af200700 1 --2- 172.21.15.27:0/1741047543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f53a80662f0 0x7f53a8114040 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:16.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.066+0000 7f53af200700 1 --2- 172.21.15.27:0/1741047543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f53a80fdd60 0x7f53a8114580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:16.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.066+0000 7f53af200700 1 -- 172.21.15.27:0/1741047543 >> 172.21.15.27:0/1741047543 conn(0x7f53a80f74c0 msgr2=0x7f53a80f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:16.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.066+0000 7f53af200700 1 -- 172.21.15.27:0/1741047543 shutdown_connections 2024-04-16T05:53:16.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.066+0000 7f53af200700 1 -- 172.21.15.27:0/1741047543 wait complete. 2024-04-16T05:53:16.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.218+0000 7f03fd800700 1 -- 172.21.15.27:0/418177788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03f80fe040 msgr2=0x7f03f80fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:16.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.218+0000 7f03fd800700 1 --2- 172.21.15.27:0/418177788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03f80fe040 0x7f03f80fe4b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f03e4004940 tx=0x7f03e4009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:16.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.218+0000 7f03fd800700 1 -- 172.21.15.27:0/418177788 shutdown_connections 2024-04-16T05:53:16.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.218+0000 7f03fd800700 1 --2- 172.21.15.27:0/418177788 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03f80fe040 0x7f03f80fe4b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:16.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.218+0000 7f03fd800700 1 --2- 172.21.15.27:0/418177788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03f80fce40 0x7f03f80fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:16.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.218+0000 7f03fd800700 1 -- 172.21.15.27:0/418177788 >> 172.21.15.27:0/418177788 conn(0x7f03f80f8530 msgr2=0x7f03f80fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:16.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.218+0000 7f03fd800700 1 -- 172.21.15.27:0/418177788 shutdown_connections 2024-04-16T05:53:16.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.218+0000 7f03fd800700 1 -- 172.21.15.27:0/418177788 wait complete. 2024-04-16T05:53:16.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.220+0000 7f03fd800700 1 Processor -- start 2024-04-16T05:53:16.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.220+0000 7f03fd800700 1 -- start start 2024-04-16T05:53:16.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.220+0000 7f03fd800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03f80fce40 0x7f03f8071e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:16.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.220+0000 7f03fd800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03f80fe040 0x7f03f8070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:16.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.220+0000 7f03fd800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f80708c0 con 0x7f03f80fe040 2024-04-16T05:53:16.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.220+0000 7f03fd800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f8070a30 con 0x7f03f80fce40 2024-04-16T05:53:16.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.221+0000 7f03f6000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03f80fe040 0x7f03f8070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:16.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.221+0000 7f03f6000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03f80fe040 0x7f03f8070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54604/0 (socket says 172.21.15.27:54604) 2024-04-16T05:53:16.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.221+0000 7f03f6000700 1 -- 172.21.15.27:0/936514196 learned_addr learned my addr 172.21.15.27:0/936514196 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:53:16.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.221+0000 7f03f6a00700 1 --2- 172.21.15.27:0/936514196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03f80fce40 0x7f03f8071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:16.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.221+0000 7f03f6000700 1 -- 172.21.15.27:0/936514196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03f80fce40 msgr2=0x7f03f8071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:16.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.221+0000 7f03f6000700 1 --2- 172.21.15.27:0/936514196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03f80fce40 0x7f03f8071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:16.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.221+0000 7f03f6000700 1 -- 172.21.15.27:0/936514196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e4005040 con 0x7f03f80fe040 2024-04-16T05:53:16.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.222+0000 7f03f6000700 1 --2- 172.21.15.27:0/936514196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03f80fe040 0x7f03f8070310 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f03e4009150 tx=0x7f03e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:16.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.222+0000 7f03eea00700 1 -- 172.21.15.27:0/936514196 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03e400e040 con 0x7f03f80fe040 2024-04-16T05:53:16.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.222+0000 7f03eea00700 1 -- 172.21.15.27:0/936514196 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03e400bbf0 con 0x7f03f80fe040 2024-04-16T05:53:16.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.222+0000 7f03fd800700 1 -- 172.21.15.27:0/936514196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f8070cb0 con 0x7f03f80fe040 2024-04-16T05:53:16.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.222+0000 7f03eea00700 1 -- 172.21.15.27:0/936514196 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03e401c630 con 0x7f03f80fe040 2024-04-16T05:53:16.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.222+0000 7f03fd800700 1 -- 172.21.15.27:0/936514196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f8115a50 con 0x7f03f80fe040 2024-04-16T05:53:16.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.224+0000 7f03fd800700 1 -- 172.21.15.27:0/936514196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f805f5f0 con 0x7f03f80fe040 2024-04-16T05:53:16.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.224+0000 7f03eea00700 1 -- 172.21.15.27:0/936514196 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f03e400bd60 con 0x7f03f80fe040 2024-04-16T05:53:16.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.225+0000 7f03eea00700 1 --2- 172.21.15.27:0/936514196 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f03dc06ec10 0x7f03dc0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:16.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.225+0000 7f03eea00700 1 -- 172.21.15.27:0/936514196 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f03e408ff70 con 0x7f03f80fe040 2024-04-16T05:53:16.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.226+0000 7f03f6a00700 1 --2- 172.21.15.27:0/936514196 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f03dc06ec10 0x7f03dc0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:16.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.226+0000 7f03f6a00700 1 --2- 172.21.15.27:0/936514196 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f03dc06ec10 0x7f03dc0710c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009fd0 tx=0x7f03e8009f00 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:16.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.230+0000 7f03eea00700 1 -- 172.21.15.27:0/936514196 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f03e4058b00 con 0x7f03f80fe040 2024-04-16T05:53:16.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.585+0000 7f03fd800700 1 -- 172.21.15.27:0/936514196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f03f8115c10 con 0x7f03f80fe040 2024-04-16T05:53:16.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.586+0000 7f03eea00700 1 -- 172.21.15.27:0/936514196 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f03e405c1c0 con 0x7f03f80fe040 2024-04-16T05:53:16.584 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:53:16.584 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:53:16.584 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:53:16.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:53:16.584 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:53:16.584 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:53:16.584 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:53:16.585 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:53:16.585 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T05:53:16.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:53:16.585 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:53:16.585 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T05:53:16.585 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:53:16.585 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:53:16.585 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 14 2024-04-16T05:53:16.585 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:53:16.585 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:53:16.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.589+0000 7f03fd800700 1 -- 172.21.15.27:0/936514196 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f03dc06ec10 msgr2=0x7f03dc0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:16.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.589+0000 7f03fd800700 1 --2- 172.21.15.27:0/936514196 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f03dc06ec10 0x7f03dc0710c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009fd0 tx=0x7f03e8009f00 comp rx=0 tx=0).stop 2024-04-16T05:53:16.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.589+0000 7f03fd800700 1 -- 172.21.15.27:0/936514196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03f80fe040 msgr2=0x7f03f8070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:16.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.589+0000 7f03fd800700 1 --2- 172.21.15.27:0/936514196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03f80fe040 0x7f03f8070310 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f03e4009150 tx=0x7f03e4009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:16.587 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.589+0000 7f03fd800700 1 -- 172.21.15.27:0/936514196 shutdown_connections 2024-04-16T05:53:16.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.589+0000 7f03fd800700 1 --2- 172.21.15.27:0/936514196 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f03dc06ec10 0x7f03dc0710c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:16.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.589+0000 7f03fd800700 1 --2- 172.21.15.27:0/936514196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03f80fce40 0x7f03f8071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:16.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.589+0000 7f03fd800700 1 --2- 172.21.15.27:0/936514196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03f80fe040 0x7f03f8070310 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:16.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.589+0000 7f03fd800700 1 -- 172.21.15.27:0/936514196 >> 172.21.15.27:0/936514196 conn(0x7f03f80f8530 msgr2=0x7f03f80f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:16.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.590+0000 7f03fd800700 1 -- 172.21.15.27:0/936514196 shutdown_connections 2024-04-16T05:53:16.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.590+0000 7f03fd800700 1 -- 172.21.15.27:0/936514196 wait complete. 2024-04-16T05:53:16.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.759+0000 7f742f000700 1 -- 172.21.15.27:0/1537925806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74280fe0e0 msgr2=0x7f74280fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:16.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.759+0000 7f742f000700 1 --2- 172.21.15.27:0/1537925806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74280fe0e0 0x7f74280fe550 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7414004940 tx=0x7f7414009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:16.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.759+0000 7f742f000700 1 -- 172.21.15.27:0/1537925806 shutdown_connections 2024-04-16T05:53:16.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.759+0000 7f742f000700 1 --2- 172.21.15.27:0/1537925806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74280fe0e0 0x7f74280fe550 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:16.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.759+0000 7f742f000700 1 --2- 172.21.15.27:0/1537925806 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74280fcef0 0x7f74280fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:16.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.759+0000 7f742f000700 1 -- 172.21.15.27:0/1537925806 >> 172.21.15.27:0/1537925806 conn(0x7f74280f8530 msgr2=0x7f74280fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:16.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.759+0000 7f742f000700 1 -- 172.21.15.27:0/1537925806 shutdown_connections 2024-04-16T05:53:16.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.759+0000 7f742f000700 1 -- 172.21.15.27:0/1537925806 wait complete. 2024-04-16T05:53:16.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.760+0000 7f742f000700 1 Processor -- start 2024-04-16T05:53:16.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.760+0000 7f742f000700 1 -- start start 2024-04-16T05:53:16.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.760+0000 7f742f000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74280fcef0 0x7f742810da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:16.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.760+0000 7f742f000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74280fe0e0 0x7f742810dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:16.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.760+0000 7f742f000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742810e5d0 con 0x7f74280fe0e0 2024-04-16T05:53:16.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.760+0000 7f742f000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742810edd0 con 0x7f74280fcef0 2024-04-16T05:53:16.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.761+0000 7f7427400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74280fe0e0 0x7f742810dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:16.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.761+0000 7f7427400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74280fe0e0 0x7f742810dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54612/0 (socket says 172.21.15.27:54612) 2024-04-16T05:53:16.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.761+0000 7f7427400700 1 -- 172.21.15.27:0/3957871681 learned_addr learned my addr 172.21.15.27:0/3957871681 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:53:16.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.761+0000 7f7427e00700 1 --2- 172.21.15.27:0/3957871681 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74280fcef0 0x7f742810da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:16.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.761+0000 7f7427400700 1 -- 172.21.15.27:0/3957871681 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74280fcef0 msgr2=0x7f742810da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:16.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.761+0000 7f7427400700 1 --2- 172.21.15.27:0/3957871681 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74280fcef0 0x7f742810da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:16.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.761+0000 7f7427400700 1 -- 172.21.15.27:0/3957871681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7414005040 con 0x7f74280fe0e0 2024-04-16T05:53:16.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.762+0000 7f7427400700 1 --2- 172.21.15.27:0/3957871681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74280fe0e0 0x7f742810dfd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f741400b3b0 tx=0x7f7414009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:16.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.762+0000 7f741fe00700 1 -- 172.21.15.27:0/3957871681 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f741400e040 con 0x7f74280fe0e0 2024-04-16T05:53:16.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.762+0000 7f741fe00700 1 -- 172.21.15.27:0/3957871681 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7414004290 con 0x7f74280fe0e0 2024-04-16T05:53:16.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.762+0000 7f742f000700 1 -- 172.21.15.27:0/3957871681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f742810f050 con 0x7f74280fe0e0 2024-04-16T05:53:16.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.762+0000 7f741fe00700 1 -- 172.21.15.27:0/3957871681 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f741401c630 con 0x7f74280fe0e0 2024-04-16T05:53:16.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.762+0000 7f742f000700 1 -- 172.21.15.27:0/3957871681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f742810f590 con 0x7f74280fe0e0 2024-04-16T05:53:16.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.763+0000 7f742f000700 1 -- 172.21.15.27:0/3957871681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f742805f5f0 con 0x7f74280fe0e0 2024-04-16T05:53:16.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.765+0000 7f741fe00700 1 -- 172.21.15.27:0/3957871681 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f741400bbf0 con 0x7f74280fe0e0 2024-04-16T05:53:16.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.766+0000 7f741fe00700 1 --2- 172.21.15.27:0/3957871681 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f740c06ec10 0x7f740c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:16.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.766+0000 7f741fe00700 1 -- 172.21.15.27:0/3957871681 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f741408ff60 con 0x7f74280fe0e0 2024-04-16T05:53:16.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.766+0000 7f7427e00700 1 --2- 172.21.15.27:0/3957871681 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f740c06ec10 0x7f740c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:16.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.767+0000 7f7427e00700 1 --2- 172.21.15.27:0/3957871681 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f740c06ec10 0x7f740c0710c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f74280fcc30 tx=0x7f7418009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:16.772 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:16.775+0000 7f741fe00700 1 -- 172.21.15.27:0/3957871681 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f741405c620 con 0x7f74280fe0e0 2024-04-16T05:53:17.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:16 smithi122 ceph-mon[80930]: from='client.34552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:53:17.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:16 smithi122 ceph-mon[80930]: pgmap v275: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:17.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:16 smithi122 ceph-mon[80930]: from='client.34556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:53:17.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:16 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:17.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:16 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:17.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:16 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/936514196' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:17.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:16 smithi027 ceph-mon[93635]: from='client.34552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:53:17.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:16 smithi027 ceph-mon[93635]: pgmap v275: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:17.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:16 smithi027 ceph-mon[93635]: from='client.34556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:53:17.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:16 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:17.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:16 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:17.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:16 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/936514196' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:17.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.072+0000 7f742f000700 1 -- 172.21.15.27:0/3957871681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f742810f910 con 0x7f74280fe0e0 2024-04-16T05:53:17.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.073+0000 7f741fe00700 1 -- 172.21.15.27:0/3957871681 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1936 (secure 0 0 0) 0x7f741405c1b0 con 0x7f74280fe0e0 2024-04-16T05:53:17.071 INFO:teuthology.orchestra.run.smithi027.stdout:e36 2024-04-16T05:53:17.071 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:53:17.071 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:53:17.071 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:53:17.071 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:53:17.071 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:53:17.071 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:53:17.071 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 35 2024-04-16T05:53:17.071 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:53:17.071 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:53:17.071 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:51:47.041625+0000 2024-04-16T05:53:17.071 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:53:17.071 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:53:17.072 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:53:17.072 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:53:17.072 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:53:17.072 INFO:teuthology.orchestra.run.smithi027.stdout:max_xattr_size 65536 2024-04-16T05:53:17.072 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:53:17.072 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:53:17.072 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 88 2024-04-16T05:53:17.072 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-16T05:53:17.072 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:53:17.072 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:53:17.073 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=34400} 2024-04-16T05:53:17.073 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:53:17.073 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:53:17.073 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:53:17.073 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:53:17.073 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:53:17.073 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:53:17.073 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:53:17.073 INFO:teuthology.orchestra.run.smithi027.stdout:bal_rank_mask -1 2024-04-16T05:53:17.073 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:53:17.073 INFO:teuthology.orchestra.run.smithi027.stdout:qdb_cluster leader: 34400 members: 34400 2024-04-16T05:53:17.073 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{0:34400} state up:active seq 23 join_fscid=1 addr [v2:172.21.15.27:6826/2831468256,v1:172.21.15.27:6827/2831468256] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:53:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:53:17.073 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:53:17.074 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:53:17.074 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:53:17.074 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{-1:34432} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6828/3219609959,v1:172.21.15.27:6829/3219609959] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:53:17.074 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:44337} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/97103507,v1:172.21.15.122:6825/97103507] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:53:17.074 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:44359} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/1030624816,v1:172.21.15.122:6827/1030624816] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:53:17.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.077+0000 7f742f000700 1 -- 172.21.15.27:0/3957871681 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f740c06ec10 msgr2=0x7f740c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:17.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.077+0000 7f742f000700 1 --2- 172.21.15.27:0/3957871681 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f740c06ec10 0x7f740c0710c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f74280fcc30 tx=0x7f7418009040 comp rx=0 tx=0).stop 2024-04-16T05:53:17.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.077+0000 7f742f000700 1 -- 172.21.15.27:0/3957871681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74280fe0e0 msgr2=0x7f742810dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:17.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.077+0000 7f742f000700 1 --2- 172.21.15.27:0/3957871681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74280fe0e0 0x7f742810dfd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f741400b3b0 tx=0x7f7414009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:17.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.077+0000 7f742f000700 1 -- 172.21.15.27:0/3957871681 shutdown_connections 2024-04-16T05:53:17.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.077+0000 7f742f000700 1 --2- 172.21.15.27:0/3957871681 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f740c06ec10 0x7f740c0710c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:17.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.077+0000 7f742f000700 1 --2- 172.21.15.27:0/3957871681 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74280fcef0 0x7f742810da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:17.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.077+0000 7f742f000700 1 --2- 172.21.15.27:0/3957871681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74280fe0e0 0x7f742810dfd0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:17.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.077+0000 7f742f000700 1 -- 172.21.15.27:0/3957871681 >> 172.21.15.27:0/3957871681 conn(0x7f74280f8530 msgr2=0x7f74280f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:17.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.077+0000 7f742f000700 1 -- 172.21.15.27:0/3957871681 shutdown_connections 2024-04-16T05:53:17.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.078+0000 7f742f000700 1 -- 172.21.15.27:0/3957871681 wait complete. 2024-04-16T05:53:17.076 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 36 2024-04-16T05:53:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.244+0000 7f3538a00700 1 -- 172.21.15.27:0/1492524257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35340662f0 msgr2=0x7f35340ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.244+0000 7f3538a00700 1 --2- 172.21.15.27:0/1492524257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35340662f0 0x7f35340ffa30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3524004940 tx=0x7f3524009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.244+0000 7f3538a00700 1 -- 172.21.15.27:0/1492524257 shutdown_connections 2024-04-16T05:53:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.245+0000 7f3538a00700 1 --2- 172.21.15.27:0/1492524257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35340fff70 0x7f3534102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.245+0000 7f3538a00700 1 --2- 172.21.15.27:0/1492524257 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35340662f0 0x7f35340ffa30 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.245+0000 7f3538a00700 1 -- 172.21.15.27:0/1492524257 >> 172.21.15.27:0/1492524257 conn(0x7f35340f74c0 msgr2=0x7f35340f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.245+0000 7f3538a00700 1 -- 172.21.15.27:0/1492524257 shutdown_connections 2024-04-16T05:53:17.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.245+0000 7f3538a00700 1 -- 172.21.15.27:0/1492524257 wait complete. 2024-04-16T05:53:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.246+0000 7f3538a00700 1 Processor -- start 2024-04-16T05:53:17.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.246+0000 7f3538a00700 1 -- start start 2024-04-16T05:53:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.247+0000 7f3538a00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35340662f0 0x7f353410fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.247+0000 7f3538a00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35340fff70 0x7f3534110190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.247+0000 7f3538a00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3534110790 con 0x7f35340fff70 2024-04-16T05:53:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.247+0000 7f3538a00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3534110fe0 con 0x7f35340662f0 2024-04-16T05:53:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.247+0000 7f3531400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35340fff70 0x7f3534110190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.247+0000 7f3531400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35340fff70 0x7f3534110190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54624/0 (socket says 172.21.15.27:54624) 2024-04-16T05:53:17.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.247+0000 7f3531400700 1 -- 172.21.15.27:0/1233168345 learned_addr learned my addr 172.21.15.27:0/1233168345 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:53:17.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.247+0000 7f3531e00700 1 --2- 172.21.15.27:0/1233168345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35340662f0 0x7f353410fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:17.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.248+0000 7f3531400700 1 -- 172.21.15.27:0/1233168345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35340662f0 msgr2=0x7f353410fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:17.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.248+0000 7f3531400700 1 --2- 172.21.15.27:0/1233168345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35340662f0 0x7f353410fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:17.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.248+0000 7f3531400700 1 -- 172.21.15.27:0/1233168345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3524005040 con 0x7f35340fff70 2024-04-16T05:53:17.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.248+0000 7f3531400700 1 --2- 172.21.15.27:0/1233168345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35340fff70 0x7f3534110190 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f352000c830 tx=0x7f3520004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.248+0000 7f352a000700 1 -- 172.21.15.27:0/1233168345 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f352000a760 con 0x7f35340fff70 2024-04-16T05:53:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.248+0000 7f3538a00700 1 -- 172.21.15.27:0/1233168345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35341112c0 con 0x7f35340fff70 2024-04-16T05:53:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.248+0000 7f352a000700 1 -- 172.21.15.27:0/1233168345 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f352000ad90 con 0x7f35340fff70 2024-04-16T05:53:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.248+0000 7f352a000700 1 -- 172.21.15.27:0/1233168345 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35200113a0 con 0x7f35340fff70 2024-04-16T05:53:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.249+0000 7f3538a00700 1 -- 172.21.15.27:0/1233168345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3534111800 con 0x7f35340fff70 2024-04-16T05:53:17.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.250+0000 7f3538a00700 1 -- 172.21.15.27:0/1233168345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35340f8ff0 con 0x7f35340fff70 2024-04-16T05:53:17.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.251+0000 7f352a000700 1 -- 172.21.15.27:0/1233168345 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3520012400 con 0x7f35340fff70 2024-04-16T05:53:17.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.252+0000 7f352a000700 1 --2- 172.21.15.27:0/1233168345 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f351806ec00 0x7f35180710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:17.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.252+0000 7f352a000700 1 -- 172.21.15.27:0/1233168345 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f352008eb90 con 0x7f35340fff70 2024-04-16T05:53:17.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.252+0000 7f3531e00700 1 --2- 172.21.15.27:0/1233168345 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f351806ec00 0x7f35180710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:17.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.253+0000 7f3531e00700 1 --2- 172.21.15.27:0/1233168345 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f351806ec00 0x7f35180710b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f352400b970 tx=0x7f3524009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:17.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.262+0000 7f352a000700 1 -- 172.21.15.27:0/1233168345 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f352005b250 con 0x7f35340fff70 2024-04-16T05:53:17.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.514+0000 7f3538a00700 1 -- 172.21.15.27:0/1233168345 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3534111a80 con 0x7f351806ec00 2024-04-16T05:53:17.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.516+0000 7f352a000700 1 -- 172.21.15.27:0/1233168345 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+440 (secure 0 0 0) 0x7f3534111a80 con 0x7f351806ec00 2024-04-16T05:53:17.515 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:53:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T05:53:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:53:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:53:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [ 2024-04-16T05:53:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "mds", 2024-04-16T05:53:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "mon", 2024-04-16T05:53:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-16T05:53:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr", 2024-04-16T05:53:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: "osd" 2024-04-16T05:53:17.515 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-16T05:53:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "16/21 daemons upgraded", 2024-04-16T05:53:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Currently upgrading alertmanager daemons", 2024-04-16T05:53:17.516 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:53:17.516 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:53:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.520+0000 7f3538a00700 1 -- 172.21.15.27:0/1233168345 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f351806ec00 msgr2=0x7f35180710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.520+0000 7f3538a00700 1 --2- 172.21.15.27:0/1233168345 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f351806ec00 0x7f35180710b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f352400b970 tx=0x7f3524009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.520+0000 7f3538a00700 1 -- 172.21.15.27:0/1233168345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35340fff70 msgr2=0x7f3534110190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.520+0000 7f3538a00700 1 --2- 172.21.15.27:0/1233168345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35340fff70 0x7f3534110190 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f352000c830 tx=0x7f3520004470 comp rx=0 tx=0).stop 2024-04-16T05:53:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.520+0000 7f3538a00700 1 -- 172.21.15.27:0/1233168345 shutdown_connections 2024-04-16T05:53:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.520+0000 7f3538a00700 1 --2- 172.21.15.27:0/1233168345 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f351806ec00 0x7f35180710b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.520+0000 7f3538a00700 1 --2- 172.21.15.27:0/1233168345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35340662f0 0x7f353410fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:17.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.520+0000 7f3538a00700 1 --2- 172.21.15.27:0/1233168345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f35340fff70 0x7f3534110190 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:17.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.520+0000 7f3538a00700 1 -- 172.21.15.27:0/1233168345 >> 172.21.15.27:0/1233168345 conn(0x7f35340f74c0 msgr2=0x7f35340fbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:17.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.520+0000 7f3538a00700 1 -- 172.21.15.27:0/1233168345 shutdown_connections 2024-04-16T05:53:17.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.521+0000 7f3538a00700 1 -- 172.21.15.27:0/1233168345 wait complete. 2024-04-16T05:53:17.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.727+0000 7fdfd4200700 1 -- 172.21.15.27:0/896856278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfcc0fcdc0 msgr2=0x7fdfcc0fd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.727+0000 7fdfd4200700 1 --2- 172.21.15.27:0/896856278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfcc0fcdc0 0x7fdfcc0fd1d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0009ce0 tx=0x7fdfc0004d10 comp rx=0 tx=0).stop 2024-04-16T05:53:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.727+0000 7fdfd4200700 1 -- 172.21.15.27:0/896856278 shutdown_connections 2024-04-16T05:53:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.727+0000 7fdfd4200700 1 --2- 172.21.15.27:0/896856278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfcc0fd710 0x7fdfcc0fe070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.727+0000 7fdfd4200700 1 --2- 172.21.15.27:0/896856278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfcc0fcdc0 0x7fdfcc0fd1d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.727+0000 7fdfd4200700 1 -- 172.21.15.27:0/896856278 >> 172.21.15.27:0/896856278 conn(0x7fdfcc0f8540 msgr2=0x7fdfcc0fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.728+0000 7fdfd4200700 1 -- 172.21.15.27:0/896856278 shutdown_connections 2024-04-16T05:53:17.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.728+0000 7fdfd4200700 1 -- 172.21.15.27:0/896856278 wait complete. 2024-04-16T05:53:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.729+0000 7fdfd4200700 1 Processor -- start 2024-04-16T05:53:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.729+0000 7fdfd4200700 1 -- start start 2024-04-16T05:53:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.729+0000 7fdfd4200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfcc0fcdc0 0x7fdfcc10dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.729+0000 7fdfd4200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfcc0fd710 0x7fdfcc10e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.729+0000 7fdfd4200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc10e860 con 0x7fdfcc0fcdc0 2024-04-16T05:53:17.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.729+0000 7fdfd4200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc10efa0 con 0x7fdfcc0fd710 2024-04-16T05:53:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.729+0000 7fdfd1800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfcc0fcdc0 0x7fdfcc10dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.730+0000 7fdfd1800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfcc0fcdc0 0x7fdfcc10dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54636/0 (socket says 172.21.15.27:54636) 2024-04-16T05:53:17.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.730+0000 7fdfd1800700 1 -- 172.21.15.27:0/788548187 learned_addr learned my addr 172.21.15.27:0/788548187 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:53:17.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.730+0000 7fdfd0e00700 1 --2- 172.21.15.27:0/788548187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfcc0fd710 0x7fdfcc10e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:17.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.730+0000 7fdfd1800700 1 -- 172.21.15.27:0/788548187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfcc0fd710 msgr2=0x7fdfcc10e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:17.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.730+0000 7fdfd1800700 1 --2- 172.21.15.27:0/788548187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfcc0fd710 0x7fdfcc10e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:17.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.730+0000 7fdfd1800700 1 -- 172.21.15.27:0/788548187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc0009940 con 0x7fdfcc0fcdc0 2024-04-16T05:53:17.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.730+0000 7fdfd1800700 1 --2- 172.21.15.27:0/788548187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfcc0fcdc0 0x7fdfcc10dd20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0009910 tx=0x7fdfc0004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:17.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.731+0000 7fdfc9600700 1 -- 172.21.15.27:0/788548187 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfc000e040 con 0x7fdfcc0fcdc0 2024-04-16T05:53:17.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.731+0000 7fdfc9600700 1 -- 172.21.15.27:0/788548187 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdfc0008ed0 con 0x7fdfcc0fcdc0 2024-04-16T05:53:17.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.731+0000 7fdfc9600700 1 -- 172.21.15.27:0/788548187 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdfc001c800 con 0x7fdfcc0fcdc0 2024-04-16T05:53:17.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.731+0000 7fdfd4200700 1 -- 172.21.15.27:0/788548187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfcc10f220 con 0x7fdfcc0fcdc0 2024-04-16T05:53:17.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.731+0000 7fdfd4200700 1 -- 172.21.15.27:0/788548187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfcc10f7e0 con 0x7fdfcc0fcdc0 2024-04-16T05:53:17.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.732+0000 7fdfd4200700 1 -- 172.21.15.27:0/788548187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfcc05f5f0 con 0x7fdfcc0fcdc0 2024-04-16T05:53:17.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.733+0000 7fdfc9600700 1 -- 172.21.15.27:0/788548187 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fdfc0012410 con 0x7fdfcc0fcdc0 2024-04-16T05:53:17.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.734+0000 7fdfc9600700 1 --2- 172.21.15.27:0/788548187 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fdfb406ec10 0x7fdfb40710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:17.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.735+0000 7fdfc9600700 1 -- 172.21.15.27:0/788548187 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7fdfc0090720 con 0x7fdfcc0fcdc0 2024-04-16T05:53:17.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.735+0000 7fdfd0e00700 1 --2- 172.21.15.27:0/788548187 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fdfb406ec10 0x7fdfb40710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:17.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.736+0000 7fdfd0e00700 1 --2- 172.21.15.27:0/788548187 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fdfb406ec10 0x7fdfb40710c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdfb8007fd0 tx=0x7fdfb8009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:17.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:17.739+0000 7fdfc9600700 1 -- 172.21.15.27:0/788548187 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fdfc005cde0 con 0x7fdfcc0fcdc0 2024-04-16T05:53:17.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:17 smithi027 ceph-mon[93635]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:53:17.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:17 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/3957871681' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:53:18.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:17 smithi122 ceph-mon[80930]: from='client.34560 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:53:18.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:17 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/3957871681' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:53:18.054 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:18.055+0000 7fdfd4200700 1 -- 172.21.15.27:0/788548187 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7fdfcc10fb10 con 0x7fdfcc0fcdc0 2024-04-16T05:53:18.055 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:18.056+0000 7fdfc9600700 1 -- 172.21.15.27:0/788548187 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7fdfc0004e10 con 0x7fdfcc0fcdc0 2024-04-16T05:53:18.055 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:53:18.057 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:18.058+0000 7fdfd4200700 1 -- 172.21.15.27:0/788548187 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fdfb406ec10 msgr2=0x7fdfb40710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:18.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:18.058+0000 7fdfd4200700 1 --2- 172.21.15.27:0/788548187 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fdfb406ec10 0x7fdfb40710c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdfb8007fd0 tx=0x7fdfb8009040 comp rx=0 tx=0).stop 2024-04-16T05:53:18.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:18.059+0000 7fdfd4200700 1 -- 172.21.15.27:0/788548187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfcc0fcdc0 msgr2=0x7fdfcc10dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:18.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:18.059+0000 7fdfd4200700 1 --2- 172.21.15.27:0/788548187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfcc0fcdc0 0x7fdfcc10dd20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0009910 tx=0x7fdfc0004d10 comp rx=0 tx=0).stop 2024-04-16T05:53:18.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:18.059+0000 7fdfd4200700 1 -- 172.21.15.27:0/788548187 shutdown_connections 2024-04-16T05:53:18.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:18.059+0000 7fdfd4200700 1 --2- 172.21.15.27:0/788548187 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fdfb406ec10 0x7fdfb40710c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:18.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:18.059+0000 7fdfd4200700 1 --2- 172.21.15.27:0/788548187 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfcc0fcdc0 0x7fdfcc10dd20 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:18.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:18.059+0000 7fdfd4200700 1 --2- 172.21.15.27:0/788548187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfcc0fd710 0x7fdfcc10e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:18.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:18.059+0000 7fdfd4200700 1 -- 172.21.15.27:0/788548187 >> 172.21.15.27:0/788548187 conn(0x7fdfcc0f8540 msgr2=0x7fdfcc0f9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:18.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:18.059+0000 7fdfd4200700 1 -- 172.21.15.27:0/788548187 shutdown_connections 2024-04-16T05:53:18.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:18.059+0000 7fdfd4200700 1 -- 172.21.15.27:0/788548187 wait complete. 2024-04-16T05:53:18.692 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:18 smithi027 ceph-mon[93635]: pgmap v276: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:18.692 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:18 smithi027 ceph-mon[93635]: from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:53:18.692 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:18 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:18.692 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:18 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:18.692 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:18 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/788548187' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:53:19.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:18 smithi122 ceph-mon[80930]: pgmap v276: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:19.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:18 smithi122 ceph-mon[80930]: from='client.34572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:53:19.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:18 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:19.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:18 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:19.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:18 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/788548187' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:53:21.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:20 smithi122 ceph-mon[80930]: pgmap v277: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:21.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:20 smithi027 ceph-mon[93635]: pgmap v277: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:21.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 590, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138693 2024-04-16T05:53:21.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 590, success, outsize: 120 2024-04-16T05:53:22.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 592, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138775 2024-04-16T05:53:22.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 592, success, outsize: 120 2024-04-16T05:53:22.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 594, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138860 2024-04-16T05:53:22.043 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 594, success, outsize: 120 2024-04-16T05:53:22.073 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 596, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 138951 2024-04-16T05:53:22.074 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 596, success, outsize: 120 2024-04-16T05:53:22.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 598, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139039 2024-04-16T05:53:22.104 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 598, success, outsize: 120 2024-04-16T05:53:22.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 600, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139121 2024-04-16T05:53:22.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 600, success, outsize: 120 2024-04-16T05:53:22.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 602, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139206 2024-04-16T05:53:22.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 602, success, outsize: 120 2024-04-16T05:53:22.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 604, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139297 2024-04-16T05:53:22.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 604, success, outsize: 120 2024-04-16T05:53:22.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 606, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139385 2024-04-16T05:53:22.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 606, success, outsize: 120 2024-04-16T05:53:22.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 608, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139467 2024-04-16T05:53:22.207 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 608, success, outsize: 120 2024-04-16T05:53:22.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 610, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139552 2024-04-16T05:53:22.224 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 610, success, outsize: 120 2024-04-16T05:53:22.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 612, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139643 2024-04-16T05:53:22.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 612, success, outsize: 120 2024-04-16T05:53:22.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 614, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139731 2024-04-16T05:53:22.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 614, success, outsize: 120 2024-04-16T05:53:22.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 616, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139813 2024-04-16T05:53:22.277 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 616, success, outsize: 120 2024-04-16T05:53:22.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 618, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139898 2024-04-16T05:53:22.296 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 618, success, outsize: 120 2024-04-16T05:53:22.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 620, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 139989 2024-04-16T05:53:22.317 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 620, success, outsize: 120 2024-04-16T05:53:22.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 622, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 140077 2024-04-16T05:53:22.338 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 622, success, outsize: 120 2024-04-16T05:53:22.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 624, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 140159 2024-04-16T05:53:22.358 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 624, success, outsize: 120 2024-04-16T05:53:22.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 626, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 140244 2024-04-16T05:53:22.381 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 626, success, outsize: 120 2024-04-16T05:53:22.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 628, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 140335 2024-04-16T05:53:22.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 628, success, outsize: 120 2024-04-16T05:53:22.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 630, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 140423 2024-04-16T05:53:22.427 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 630, success, outsize: 120 2024-04-16T05:53:22.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 632, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 140505 2024-04-16T05:53:22.450 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 632, success, outsize: 120 2024-04-16T05:53:22.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 634, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 140590 2024-04-16T05:53:22.474 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 634, success, outsize: 120 2024-04-16T05:53:22.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 636, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 140681 2024-04-16T05:53:22.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 636, success, outsize: 120 2024-04-16T05:53:22.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 638, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 140769 2024-04-16T05:53:22.526 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 638, success, outsize: 120 2024-04-16T05:53:22.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 640, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 140851 2024-04-16T05:53:22.552 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 640, success, outsize: 120 2024-04-16T05:53:22.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 642, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 140936 2024-04-16T05:53:22.576 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 642, success, outsize: 120 2024-04-16T05:53:22.601 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 644, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 141027 2024-04-16T05:53:22.601 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 644, success, outsize: 120 2024-04-16T05:53:22.981 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:22 smithi027 ceph-mon[93635]: pgmap v278: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:23.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:22 smithi122 ceph-mon[80930]: pgmap v278: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:24.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: pgmap v279: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:24.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:24.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:24.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:53:24.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:53:24.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:24.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T05:53:24.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:53:24.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:24 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.825 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: pgmap v279: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:24.825 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:24.825 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:24.825 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:53:24.825 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:53:24.825 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:24.825 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T05:53:24.825 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:53:24.826 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.826 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.826 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.826 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.826 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.826 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.826 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.826 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.826 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.826 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.826 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.826 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.826 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.826 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:24.826 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:24 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:25.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:25 smithi027 ceph-mon[93635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T05:53:26.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:25 smithi122 ceph-mon[80930]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-16T05:53:27.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:26 smithi122 ceph-mon[80930]: pgmap v280: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:27.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:26 smithi027 ceph-mon[93635]: pgmap v280: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:28.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:27 smithi122 ceph-mon[80930]: Upgrade: Updating grafana.smithi027 2024-04-16T05:53:28.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:27 smithi122 ceph-mon[80930]: Deploying daemon grafana.smithi027 on smithi027 2024-04-16T05:53:28.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:27 smithi027 ceph-mon[93635]: Upgrade: Updating grafana.smithi027 2024-04-16T05:53:28.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:27 smithi027 ceph-mon[93635]: Deploying daemon grafana.smithi027 on smithi027 2024-04-16T05:53:28.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:28 smithi027 ceph-mon[93635]: pgmap v281: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:29.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:28 smithi122 ceph-mon[80930]: pgmap v281: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:30.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:53:30.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:53:31.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:30 smithi122 ceph-mon[80930]: pgmap v282: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:31.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:30 smithi027 ceph-mon[93635]: pgmap v282: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:33.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:32 smithi122 ceph-mon[80930]: pgmap v283: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:33.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:32 smithi027 ceph-mon[93635]: pgmap v283: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:35.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:34 smithi122 ceph-mon[80930]: pgmap v284: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:35.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:34 smithi027 ceph-mon[93635]: pgmap v284: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:36.643 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:36 smithi027 ceph-mon[93635]: pgmap v285: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:37.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:36 smithi122 ceph-mon[80930]: pgmap v285: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:38.647 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:38 smithi027 ceph-mon[93635]: pgmap v286: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:39.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:38 smithi122 ceph-mon[80930]: pgmap v286: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:40.947 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:40 smithi027 ceph-mon[93635]: pgmap v287: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:41.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:40 smithi122 ceph-mon[80930]: pgmap v287: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:42.911 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:42 smithi027 ceph-mon[93635]: pgmap v288: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:43.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:42 smithi122 ceph-mon[80930]: pgmap v288: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:44.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:44 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:44.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:44 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:44.521 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:44 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:53:44.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:44 smithi122 ceph-mon[80930]: pgmap v289: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:44.522 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:44 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:53:44.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:44.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:44.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:53:44.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:44 smithi027 ceph-mon[93635]: pgmap v289: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:44.557 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:53:46.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:46 smithi122 ceph-mon[80930]: pgmap v290: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:46.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:46 smithi027 ceph-mon[93635]: pgmap v290: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:48.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.229+0000 7fb6cfc00700 1 -- 172.21.15.27:0/1571627241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6c8103420 msgr2=0x7fb6c8103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:48.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.229+0000 7fb6cfc00700 1 --2- 172.21.15.27:0/1571627241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6c8103420 0x7fb6c8103830 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc004940 tx=0x7fb6bc009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:48.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.229+0000 7fb6cfc00700 1 -- 172.21.15.27:0/1571627241 shutdown_connections 2024-04-16T05:53:48.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.229+0000 7fb6cfc00700 1 --2- 172.21.15.27:0/1571627241 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6c81046d0 0x7fb6c8104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:48.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.229+0000 7fb6cfc00700 1 --2- 172.21.15.27:0/1571627241 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6c8103420 0x7fb6c8103830 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:48.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.229+0000 7fb6cfc00700 1 -- 172.21.15.27:0/1571627241 >> 172.21.15.27:0/1571627241 conn(0x7fb6c8072640 msgr2=0x7fb6c8074a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:48.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.229+0000 7fb6cfc00700 1 -- 172.21.15.27:0/1571627241 shutdown_connections 2024-04-16T05:53:48.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.229+0000 7fb6cfc00700 1 -- 172.21.15.27:0/1571627241 wait complete. 2024-04-16T05:53:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.230+0000 7fb6cfc00700 1 Processor -- start 2024-04-16T05:53:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.230+0000 7fb6cfc00700 1 -- start start 2024-04-16T05:53:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.231+0000 7fb6cfc00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6c8103420 0x7fb6c8113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.231+0000 7fb6cfc00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6c81046d0 0x7fb6c81144c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.231+0000 7fb6cfc00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c8114ac0 con 0x7fb6c8103420 2024-04-16T05:53:48.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.231+0000 7fb6cfc00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c8114c30 con 0x7fb6c81046d0 2024-04-16T05:53:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.231+0000 7fb6cd200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6c8103420 0x7fb6c8113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.231+0000 7fb6cd200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6c8103420 0x7fb6c8113f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59576/0 (socket says 172.21.15.27:59576) 2024-04-16T05:53:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.231+0000 7fb6cd200700 1 -- 172.21.15.27:0/537392564 learned_addr learned my addr 172.21.15.27:0/537392564 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:53:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.231+0000 7fb6c7e00700 1 --2- 172.21.15.27:0/537392564 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6c81046d0 0x7fb6c81144c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.231+0000 7fb6cd200700 1 -- 172.21.15.27:0/537392564 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6c81046d0 msgr2=0x7fb6c81144c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.231+0000 7fb6cd200700 1 --2- 172.21.15.27:0/537392564 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6c81046d0 0x7fb6c81144c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.231+0000 7fb6cd200700 1 -- 172.21.15.27:0/537392564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc005040 con 0x7fb6c8103420 2024-04-16T05:53:48.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.231+0000 7fb6c7e00700 1 --2- 172.21.15.27:0/537392564 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6c81046d0 0x7fb6c81144c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:53:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.232+0000 7fb6cd200700 1 --2- 172.21.15.27:0/537392564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6c8103420 0x7fb6c8113f80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc00b3b0 tx=0x7fb6bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.232+0000 7fb6c4c00700 1 -- 172.21.15.27:0/537392564 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6bc00e040 con 0x7fb6c8103420 2024-04-16T05:53:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.232+0000 7fb6c4c00700 1 -- 172.21.15.27:0/537392564 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb6bc004290 con 0x7fb6c8103420 2024-04-16T05:53:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.232+0000 7fb6c4c00700 1 -- 172.21.15.27:0/537392564 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6bc01c630 con 0x7fb6c8103420 2024-04-16T05:53:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.232+0000 7fb6cfc00700 1 -- 172.21.15.27:0/537392564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c8119630 con 0x7fb6c8103420 2024-04-16T05:53:48.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.232+0000 7fb6cfc00700 1 -- 172.21.15.27:0/537392564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c8119bf0 con 0x7fb6c8103420 2024-04-16T05:53:48.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.233+0000 7fb6cfc00700 1 -- 172.21.15.27:0/537392564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6c804c710 con 0x7fb6c8103420 2024-04-16T05:53:48.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.235+0000 7fb6c4c00700 1 -- 172.21.15.27:0/537392564 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb6bc00bbf0 con 0x7fb6c8103420 2024-04-16T05:53:48.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.236+0000 7fb6c4c00700 1 --2- 172.21.15.27:0/537392564 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb6b006ec00 0x7fb6b00710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:48.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.236+0000 7fb6c4c00700 1 -- 172.21.15.27:0/537392564 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7fb6bc08fdd0 con 0x7fb6c8103420 2024-04-16T05:53:48.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.236+0000 7fb6c7e00700 1 --2- 172.21.15.27:0/537392564 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb6b006ec00 0x7fb6b00710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:48.234 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.237+0000 7fb6c7e00700 1 --2- 172.21.15.27:0/537392564 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb6b006ec00 0x7fb6b00710b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8009fd0 tx=0x7fb6b800b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:48.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.241+0000 7fb6c4c00700 1 -- 172.21.15.27:0/537392564 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb6bc05c5d0 con 0x7fb6c8103420 2024-04-16T05:53:48.508 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.510+0000 7fb6cfc00700 1 -- 172.21.15.27:0/537392564 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6c8108a60 con 0x7fb6b006ec00 2024-04-16T05:53:48.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.513+0000 7fb6c4c00700 1 -- 172.21.15.27:0/537392564 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+435 (secure 0 0 0) 0x7fb6c8108a60 con 0x7fb6b006ec00 2024-04-16T05:53:48.514 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.516+0000 7fb6cfc00700 1 -- 172.21.15.27:0/537392564 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb6b006ec00 msgr2=0x7fb6b00710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:48.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.516+0000 7fb6cfc00700 1 --2- 172.21.15.27:0/537392564 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb6b006ec00 0x7fb6b00710b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb6b8009fd0 tx=0x7fb6b800b040 comp rx=0 tx=0).stop 2024-04-16T05:53:48.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.516+0000 7fb6cfc00700 1 -- 172.21.15.27:0/537392564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6c8103420 msgr2=0x7fb6c8113f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:48.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.516+0000 7fb6cfc00700 1 --2- 172.21.15.27:0/537392564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6c8103420 0x7fb6c8113f80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc00b3b0 tx=0x7fb6bc009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:48.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.516+0000 7fb6cfc00700 1 -- 172.21.15.27:0/537392564 shutdown_connections 2024-04-16T05:53:48.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.516+0000 7fb6cfc00700 1 --2- 172.21.15.27:0/537392564 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb6b006ec00 0x7fb6b00710b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:48.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.516+0000 7fb6cfc00700 1 --2- 172.21.15.27:0/537392564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb6c8103420 0x7fb6c8113f80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:48.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.516+0000 7fb6cfc00700 1 --2- 172.21.15.27:0/537392564 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6c81046d0 0x7fb6c81144c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:48.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.516+0000 7fb6cfc00700 1 -- 172.21.15.27:0/537392564 >> 172.21.15.27:0/537392564 conn(0x7fb6c8072640 msgr2=0x7fb6c8073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:48.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.516+0000 7fb6cfc00700 1 -- 172.21.15.27:0/537392564 shutdown_connections 2024-04-16T05:53:48.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.516+0000 7fb6cfc00700 1 -- 172.21.15.27:0/537392564 wait complete. 2024-04-16T05:53:48.530 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:53:48.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.676+0000 7f5c8c800700 1 -- 172.21.15.27:0/384126583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c840ff000 msgr2=0x7f5c840ff410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:48.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.676+0000 7f5c8c800700 1 --2- 172.21.15.27:0/384126583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c840ff000 0x7f5c840ff410 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5c78004940 tx=0x7f5c78009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:48.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.676+0000 7f5c8c800700 1 -- 172.21.15.27:0/384126583 shutdown_connections 2024-04-16T05:53:48.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.676+0000 7f5c8c800700 1 --2- 172.21.15.27:0/384126583 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c841002b0 0x7f5c84100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:48.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.676+0000 7f5c8c800700 1 --2- 172.21.15.27:0/384126583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c840ff000 0x7f5c840ff410 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:48.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.676+0000 7f5c8c800700 1 -- 172.21.15.27:0/384126583 >> 172.21.15.27:0/384126583 conn(0x7f5c840fa950 msgr2=0x7f5c840fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:48.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.676+0000 7f5c8c800700 1 -- 172.21.15.27:0/384126583 shutdown_connections 2024-04-16T05:53:48.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.676+0000 7f5c8c800700 1 -- 172.21.15.27:0/384126583 wait complete. 2024-04-16T05:53:48.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.677+0000 7f5c8c800700 1 Processor -- start 2024-04-16T05:53:48.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.677+0000 7f5c8c800700 1 -- start start 2024-04-16T05:53:48.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.678+0000 7f5c8c800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c840ff000 0x7f5c8410fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:48.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.678+0000 7f5c8c800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c841002b0 0x7f5c84110160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:48.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.678+0000 7f5c8c800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c84110760 con 0x7f5c840ff000 2024-04-16T05:53:48.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.678+0000 7f5c8c800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c84110fb0 con 0x7f5c841002b0 2024-04-16T05:53:48.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.678+0000 7f5c89e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c840ff000 0x7f5c8410fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:48.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.678+0000 7f5c89e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c840ff000 0x7f5c8410fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59592/0 (socket says 172.21.15.27:59592) 2024-04-16T05:53:48.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.678+0000 7f5c89e00700 1 -- 172.21.15.27:0/3674224863 learned_addr learned my addr 172.21.15.27:0/3674224863 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:53:48.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.678+0000 7f5c89400700 1 --2- 172.21.15.27:0/3674224863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c841002b0 0x7f5c84110160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:48.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.679+0000 7f5c89e00700 1 -- 172.21.15.27:0/3674224863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c841002b0 msgr2=0x7f5c84110160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:48.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.679+0000 7f5c89e00700 1 --2- 172.21.15.27:0/3674224863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c841002b0 0x7f5c84110160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:48.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.679+0000 7f5c89e00700 1 -- 172.21.15.27:0/3674224863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c78005040 con 0x7f5c840ff000 2024-04-16T05:53:48.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.679+0000 7f5c89e00700 1 --2- 172.21.15.27:0/3674224863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c840ff000 0x7f5c8410fc20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5c78009150 tx=0x7f5c78009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:48.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.679+0000 7f5c7e000700 1 -- 172.21.15.27:0/3674224863 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c7800e040 con 0x7f5c840ff000 2024-04-16T05:53:48.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.679+0000 7f5c7e000700 1 -- 172.21.15.27:0/3674224863 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c7800bbf0 con 0x7f5c840ff000 2024-04-16T05:53:48.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.679+0000 7f5c7e000700 1 -- 172.21.15.27:0/3674224863 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c7801c630 con 0x7f5c840ff000 2024-04-16T05:53:48.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.679+0000 7f5c8c800700 1 -- 172.21.15.27:0/3674224863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c84111230 con 0x7f5c840ff000 2024-04-16T05:53:48.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.680+0000 7f5c8c800700 1 -- 172.21.15.27:0/3674224863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c84111770 con 0x7f5c840ff000 2024-04-16T05:53:48.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.681+0000 7f5c8c800700 1 -- 172.21.15.27:0/3674224863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c8405f5f0 con 0x7f5c840ff000 2024-04-16T05:53:48.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.682+0000 7f5c7e000700 1 -- 172.21.15.27:0/3674224863 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f5c7800bd60 con 0x7f5c840ff000 2024-04-16T05:53:48.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.683+0000 7f5c7e000700 1 --2- 172.21.15.27:0/3674224863 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5c6c06ec10 0x7f5c6c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:48.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.683+0000 7f5c7e000700 1 -- 172.21.15.27:0/3674224863 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f5c7808fe30 con 0x7f5c840ff000 2024-04-16T05:53:48.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.683+0000 7f5c89400700 1 --2- 172.21.15.27:0/3674224863 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5c6c06ec10 0x7f5c6c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:48.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.684+0000 7f5c89400700 1 --2- 172.21.15.27:0/3674224863 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5c6c06ec10 0x7f5c6c0710c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5c74007fd0 tx=0x7f5c74009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:48.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.688+0000 7f5c7e000700 1 -- 172.21.15.27:0/3674224863 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f5c7805c570 con 0x7f5c840ff000 2024-04-16T05:53:48.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:48 smithi122 ceph-mon[80930]: pgmap v291: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:48.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:48 smithi027 ceph-mon[93635]: pgmap v291: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:48.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.975+0000 7f5c8c800700 1 -- 172.21.15.27:0/3674224863 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c84111b10 con 0x7f5c6c06ec10 2024-04-16T05:53:48.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.977+0000 7f5c7e000700 1 -- 172.21.15.27:0/3674224863 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+435 (secure 0 0 0) 0x7f5c84111b10 con 0x7f5c6c06ec10 2024-04-16T05:53:48.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.980+0000 7f5c8c800700 1 -- 172.21.15.27:0/3674224863 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5c6c06ec10 msgr2=0x7f5c6c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:48.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.980+0000 7f5c8c800700 1 --2- 172.21.15.27:0/3674224863 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5c6c06ec10 0x7f5c6c0710c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5c74007fd0 tx=0x7f5c74009040 comp rx=0 tx=0).stop 2024-04-16T05:53:48.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.980+0000 7f5c8c800700 1 -- 172.21.15.27:0/3674224863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c840ff000 msgr2=0x7f5c8410fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:48.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.980+0000 7f5c8c800700 1 --2- 172.21.15.27:0/3674224863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c840ff000 0x7f5c8410fc20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5c78009150 tx=0x7f5c78009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:48.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.980+0000 7f5c8c800700 1 -- 172.21.15.27:0/3674224863 shutdown_connections 2024-04-16T05:53:48.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.980+0000 7f5c8c800700 1 --2- 172.21.15.27:0/3674224863 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f5c6c06ec10 0x7f5c6c0710c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:48.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.980+0000 7f5c8c800700 1 --2- 172.21.15.27:0/3674224863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c840ff000 0x7f5c8410fc20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:48.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.980+0000 7f5c8c800700 1 --2- 172.21.15.27:0/3674224863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5c841002b0 0x7f5c84110160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:48.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.980+0000 7f5c8c800700 1 -- 172.21.15.27:0/3674224863 >> 172.21.15.27:0/3674224863 conn(0x7f5c840fa950 msgr2=0x7f5c840fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:48.979 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.980+0000 7f5c8c800700 1 -- 172.21.15.27:0/3674224863 shutdown_connections 2024-04-16T05:53:48.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:48.980+0000 7f5c8c800700 1 -- 172.21.15.27:0/3674224863 wait complete. 2024-04-16T05:53:49.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.131+0000 7f716ea00700 1 -- 172.21.15.27:0/1233653342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71681002d0 msgr2=0x7f7168100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:49.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.131+0000 7f716ea00700 1 --2- 172.21.15.27:0/1233653342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71681002d0 0x7f7168100740 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7150004940 tx=0x7f7150009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:49.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.131+0000 7f716ea00700 1 -- 172.21.15.27:0/1233653342 shutdown_connections 2024-04-16T05:53:49.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.131+0000 7f716ea00700 1 --2- 172.21.15.27:0/1233653342 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71681002d0 0x7f7168100740 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:49.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.131+0000 7f716ea00700 1 --2- 172.21.15.27:0/1233653342 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71680ff020 0x7f71680ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:49.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.131+0000 7f716ea00700 1 -- 172.21.15.27:0/1233653342 >> 172.21.15.27:0/1233653342 conn(0x7f71680fa950 msgr2=0x7f71680fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:49.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.131+0000 7f716ea00700 1 -- 172.21.15.27:0/1233653342 shutdown_connections 2024-04-16T05:53:49.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.131+0000 7f716ea00700 1 -- 172.21.15.27:0/1233653342 wait complete. 2024-04-16T05:53:49.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.132+0000 7f716ea00700 1 Processor -- start 2024-04-16T05:53:49.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.132+0000 7f716ea00700 1 -- start start 2024-04-16T05:53:49.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.133+0000 7f716ea00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71680ff020 0x7f716810fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:49.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.133+0000 7f716ea00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71681002d0 0x7f71681100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:49.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.133+0000 7f716ea00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71681106b0 con 0x7f71680ff020 2024-04-16T05:53:49.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.133+0000 7f716ea00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7168110820 con 0x7f71681002d0 2024-04-16T05:53:49.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.133+0000 7f7167e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71680ff020 0x7f716810fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:49.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.133+0000 7f7167e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71680ff020 0x7f716810fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59604/0 (socket says 172.21.15.27:59604) 2024-04-16T05:53:49.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.133+0000 7f7167e00700 1 -- 172.21.15.27:0/455823448 learned_addr learned my addr 172.21.15.27:0/455823448 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:53:49.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.133+0000 7f715fe00700 1 --2- 172.21.15.27:0/455823448 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71681002d0 0x7f71681100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:49.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.133+0000 7f7167e00700 1 -- 172.21.15.27:0/455823448 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71681002d0 msgr2=0x7f71681100b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:49.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.133+0000 7f7167e00700 1 --2- 172.21.15.27:0/455823448 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71681002d0 0x7f71681100b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:49.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.133+0000 7f7167e00700 1 -- 172.21.15.27:0/455823448 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7150005040 con 0x7f71680ff020 2024-04-16T05:53:49.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.134+0000 7f7167e00700 1 --2- 172.21.15.27:0/455823448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71680ff020 0x7f716810fb70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7158002a50 tx=0x7f71580046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:49.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.134+0000 7f7164c00700 1 -- 172.21.15.27:0/455823448 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7158007cc0 con 0x7f71680ff020 2024-04-16T05:53:49.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.134+0000 7f7164c00700 1 -- 172.21.15.27:0/455823448 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f715800f040 con 0x7f71680ff020 2024-04-16T05:53:49.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.134+0000 7f7164c00700 1 -- 172.21.15.27:0/455823448 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7158009640 con 0x7f71680ff020 2024-04-16T05:53:49.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.134+0000 7f716ea00700 1 -- 172.21.15.27:0/455823448 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7168115280 con 0x7f71680ff020 2024-04-16T05:53:49.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.134+0000 7f716ea00700 1 -- 172.21.15.27:0/455823448 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7168115840 con 0x7f71680ff020 2024-04-16T05:53:49.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.135+0000 7f716ea00700 1 -- 172.21.15.27:0/455823448 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f716804c710 con 0x7f71680ff020 2024-04-16T05:53:49.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.136+0000 7f7164c00700 1 -- 172.21.15.27:0/455823448 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f71580097a0 con 0x7f71680ff020 2024-04-16T05:53:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.137+0000 7f7164c00700 1 --2- 172.21.15.27:0/455823448 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f714c06ec00 0x7f714c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.137+0000 7f7164c00700 1 -- 172.21.15.27:0/455823448 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f715808e850 con 0x7f71680ff020 2024-04-16T05:53:49.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.138+0000 7f715fe00700 1 --2- 172.21.15.27:0/455823448 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f714c06ec00 0x7f714c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:49.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.138+0000 7f715fe00700 1 --2- 172.21.15.27:0/455823448 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f714c06ec00 0x7f714c0710b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f715000b970 tx=0x7f7150009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:49.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.142+0000 7f7164c00700 1 -- 172.21.15.27:0/455823448 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f715805af10 con 0x7f71680ff020 2024-04-16T05:53:49.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.390+0000 7f716ea00700 1 -- 172.21.15.27:0/455823448 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f71681045f0 con 0x7f714c06ec00 2024-04-16T05:53:49.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.420+0000 7f7164c00700 1 -- 172.21.15.27:0/455823448 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f71681045f0 con 0x7f714c06ec00 2024-04-16T05:53:49.418 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:53:49.419 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (42s) 33s ago 20m 13.7M - 0.25.0 c8568f914cd2 42321467ded6 2024-04-16T05:53:49.419 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (6m) 33s ago 21m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 4aa1f0006685 2024-04-16T05:53:49.419 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (6m) 89s ago 19m 6656k - 19.0.0-2884-g218a6914 a4e4811815e4 1c1912507d56 2024-04-16T05:53:49.419 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 starting - - - - 2024-04-16T05:53:49.419 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (2m) 33s ago 13m 11.8M - 19.0.0-2884-g218a6914 a4e4811815e4 b3c2178c537e 2024-04-16T05:53:49.419 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (3m) 33s ago 13m 19.4M - 19.0.0-2884-g218a6914 a4e4811815e4 14b0c83f2891 2024-04-16T05:53:49.419 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (2m) 89s ago 13m 11.2M - 19.0.0-2884-g218a6914 a4e4811815e4 8ba95adfaaef 2024-04-16T05:53:49.419 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (118s) 89s ago 13m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 c60e9ca0f1c8 2024-04-16T05:53:49.419 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:8443,9283,8765 running (9m) 33s ago 22m 514M - 19.0.0-2884-g218a6914 a4e4811815e4 b69a0b08e12b 2024-04-16T05:53:49.419 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283,8765 running (8m) 89s ago 19m 454M - 19.0.0-2884-g218a6914 a4e4811815e4 32e27bcd7e42 2024-04-16T05:53:49.419 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (8m) 33s ago 22m 54.5M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 299ca337e4ca 2024-04-16T05:53:49.419 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (7m) 89s ago 19m 40.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5db5f3037ed0 2024-04-16T05:53:49.419 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (100s) 33s ago 21m 7944k - 1.5.0 0da6a335fe13 f3382c6beacb 2024-04-16T05:53:49.419 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (93s) 89s ago 19m 3078k - 1.5.0 0da6a335fe13 1f6817821f3a 2024-04-16T05:53:49.420 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (6m) 33s ago 18m 47.2M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 cef03b67bc5a 2024-04-16T05:53:49.420 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (5m) 33s ago 17m 46.1M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3a6ab1fc6e79 2024-04-16T05:53:49.420 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (5m) 33s ago 17m 44.7M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3524731d7277 2024-04-16T05:53:49.420 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (4m) 89s ago 16m 44.7M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 897d3579f929 2024-04-16T05:53:49.420 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (4m) 89s ago 15m 42.2M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 fa97758ff7c5 2024-04-16T05:53:49.420 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (3m) 89s ago 15m 49.6M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 fd12260a820d 2024-04-16T05:53:49.420 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (67s) 33s ago 20m 47.6M - 2.43.0 a07b618ecd1d a6cddf5443d5 2024-04-16T05:53:49.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.424+0000 7f716ea00700 1 -- 172.21.15.27:0/455823448 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f714c06ec00 msgr2=0x7f714c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:49.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.424+0000 7f716ea00700 1 --2- 172.21.15.27:0/455823448 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f714c06ec00 0x7f714c0710b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f715000b970 tx=0x7f7150009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:49.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.424+0000 7f716ea00700 1 -- 172.21.15.27:0/455823448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71680ff020 msgr2=0x7f716810fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:49.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.424+0000 7f716ea00700 1 --2- 172.21.15.27:0/455823448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71680ff020 0x7f716810fb70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7158002a50 tx=0x7f71580046b0 comp rx=0 tx=0).stop 2024-04-16T05:53:49.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.424+0000 7f716ea00700 1 -- 172.21.15.27:0/455823448 shutdown_connections 2024-04-16T05:53:49.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.424+0000 7f716ea00700 1 --2- 172.21.15.27:0/455823448 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f714c06ec00 0x7f714c0710b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:49.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.424+0000 7f716ea00700 1 --2- 172.21.15.27:0/455823448 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71680ff020 0x7f716810fb70 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:49.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.424+0000 7f716ea00700 1 --2- 172.21.15.27:0/455823448 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71681002d0 0x7f71681100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:49.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.424+0000 7f716ea00700 1 -- 172.21.15.27:0/455823448 >> 172.21.15.27:0/455823448 conn(0x7f71680fa950 msgr2=0x7f71680fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:49.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.424+0000 7f716ea00700 1 -- 172.21.15.27:0/455823448 shutdown_connections 2024-04-16T05:53:49.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.424+0000 7f716ea00700 1 -- 172.21.15.27:0/455823448 wait complete. 2024-04-16T05:53:49.556 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:49 smithi027 ceph-mon[93635]: from='client.34580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:53:49.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.579+0000 7f42ed400700 1 -- 172.21.15.27:0/411229228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42e80662f0 msgr2=0x7f42e8101e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:49.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.579+0000 7f42ed400700 1 --2- 172.21.15.27:0/411229228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42e80662f0 0x7f42e8101e30 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f42d8004940 tx=0x7f42d8009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:49.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.579+0000 7f42ed400700 1 -- 172.21.15.27:0/411229228 shutdown_connections 2024-04-16T05:53:49.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.579+0000 7f42ed400700 1 --2- 172.21.15.27:0/411229228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42e8102370 0x7f42e8104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:49.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.579+0000 7f42ed400700 1 --2- 172.21.15.27:0/411229228 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42e80662f0 0x7f42e8101e30 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:49.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.579+0000 7f42ed400700 1 -- 172.21.15.27:0/411229228 >> 172.21.15.27:0/411229228 conn(0x7f42e80f74c0 msgr2=0x7f42e80f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:49.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.579+0000 7f42ed400700 1 -- 172.21.15.27:0/411229228 shutdown_connections 2024-04-16T05:53:49.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.579+0000 7f42ed400700 1 -- 172.21.15.27:0/411229228 wait complete. 2024-04-16T05:53:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.580+0000 7f42ed400700 1 Processor -- start 2024-04-16T05:53:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.580+0000 7f42ed400700 1 -- start start 2024-04-16T05:53:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.580+0000 7f42ed400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42e80662f0 0x7f42e810fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.580+0000 7f42ed400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42e8102370 0x7f42e81101d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.580+0000 7f42ed400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e8110860 con 0x7f42e80662f0 2024-04-16T05:53:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.580+0000 7f42ed400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e8111000 con 0x7f42e8102370 2024-04-16T05:53:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.580+0000 7f42e6800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42e80662f0 0x7f42e810fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.580+0000 7f42e6800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42e80662f0 0x7f42e810fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59618/0 (socket says 172.21.15.27:59618) 2024-04-16T05:53:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.580+0000 7f42e6800700 1 -- 172.21.15.27:0/1686467972 learned_addr learned my addr 172.21.15.27:0/1686467972 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:53:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.581+0000 7f42dfe00700 1 --2- 172.21.15.27:0/1686467972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42e8102370 0x7f42e81101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:49.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.581+0000 7f42e6800700 1 -- 172.21.15.27:0/1686467972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42e8102370 msgr2=0x7f42e81101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:49.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.581+0000 7f42e6800700 1 --2- 172.21.15.27:0/1686467972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42e8102370 0x7f42e81101d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:49.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.581+0000 7f42e6800700 1 -- 172.21.15.27:0/1686467972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d8005040 con 0x7f42e80662f0 2024-04-16T05:53:49.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.581+0000 7f42e6800700 1 --2- 172.21.15.27:0/1686467972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42e80662f0 0x7f42e810fc90 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f42d800b970 tx=0x7f42d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:49.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.582+0000 7f42dea00700 1 -- 172.21.15.27:0/1686467972 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42d800e040 con 0x7f42e80662f0 2024-04-16T05:53:49.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.582+0000 7f42dea00700 1 -- 172.21.15.27:0/1686467972 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42d8004290 con 0x7f42e80662f0 2024-04-16T05:53:49.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.582+0000 7f42dea00700 1 -- 172.21.15.27:0/1686467972 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42d801c630 con 0x7f42e80662f0 2024-04-16T05:53:49.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.582+0000 7f42ed400700 1 -- 172.21.15.27:0/1686467972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e8111280 con 0x7f42e80662f0 2024-04-16T05:53:49.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.582+0000 7f42ed400700 1 -- 172.21.15.27:0/1686467972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e8111840 con 0x7f42e80662f0 2024-04-16T05:53:49.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.583+0000 7f42ed400700 1 -- 172.21.15.27:0/1686467972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e805f5f0 con 0x7f42e80662f0 2024-04-16T05:53:49.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.584+0000 7f42dea00700 1 -- 172.21.15.27:0/1686467972 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f42d800bbf0 con 0x7f42e80662f0 2024-04-16T05:53:49.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.585+0000 7f42dea00700 1 --2- 172.21.15.27:0/1686467972 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f42cc06ec10 0x7f42cc0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:49.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.585+0000 7f42dea00700 1 -- 172.21.15.27:0/1686467972 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f42d808fd00 con 0x7f42e80662f0 2024-04-16T05:53:49.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.585+0000 7f42dfe00700 1 --2- 172.21.15.27:0/1686467972 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f42cc06ec10 0x7f42cc0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:49.583 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.586+0000 7f42dfe00700 1 --2- 172.21.15.27:0/1686467972 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f42cc06ec10 0x7f42cc0710c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f42d0006fd0 tx=0x7f42d0008040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:49.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.591+0000 7f42dea00700 1 -- 172.21.15.27:0/1686467972 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f42d8093030 con 0x7f42e80662f0 2024-04-16T05:53:49.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:49 smithi122 ceph-mon[80930]: from='client.34580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:53:49.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.923+0000 7f42ed400700 1 -- 172.21.15.27:0/1686467972 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f42e8107b30 con 0x7f42e80662f0 2024-04-16T05:53:49.922 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.925+0000 7f42dea00700 1 -- 172.21.15.27:0/1686467972 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f42d8013e60 con 0x7f42e80662f0 2024-04-16T05:53:49.922 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:53:49.923 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:53:49.923 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:53:49.923 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:53:49.923 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:53:49.923 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:53:49.923 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:53:49.923 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:53:49.923 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T05:53:49.923 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:53:49.923 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:53:49.923 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T05:53:49.923 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:53:49.923 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:53:49.924 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 14 2024-04-16T05:53:49.924 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:53:49.924 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:53:49.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.928+0000 7f42ed400700 1 -- 172.21.15.27:0/1686467972 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f42cc06ec10 msgr2=0x7f42cc0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:49.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.928+0000 7f42ed400700 1 --2- 172.21.15.27:0/1686467972 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f42cc06ec10 0x7f42cc0710c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f42d0006fd0 tx=0x7f42d0008040 comp rx=0 tx=0).stop 2024-04-16T05:53:49.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.928+0000 7f42ed400700 1 -- 172.21.15.27:0/1686467972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42e80662f0 msgr2=0x7f42e810fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:49.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.928+0000 7f42ed400700 1 --2- 172.21.15.27:0/1686467972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42e80662f0 0x7f42e810fc90 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f42d800b970 tx=0x7f42d8009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:49.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.928+0000 7f42ed400700 1 -- 172.21.15.27:0/1686467972 shutdown_connections 2024-04-16T05:53:49.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.928+0000 7f42ed400700 1 --2- 172.21.15.27:0/1686467972 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f42cc06ec10 0x7f42cc0710c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:49.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.928+0000 7f42ed400700 1 --2- 172.21.15.27:0/1686467972 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42e80662f0 0x7f42e810fc90 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:49.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.928+0000 7f42ed400700 1 --2- 172.21.15.27:0/1686467972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42e8102370 0x7f42e81101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:49.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.928+0000 7f42ed400700 1 -- 172.21.15.27:0/1686467972 >> 172.21.15.27:0/1686467972 conn(0x7f42e80f74c0 msgr2=0x7f42e80f7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:49.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.928+0000 7f42ed400700 1 -- 172.21.15.27:0/1686467972 shutdown_connections 2024-04-16T05:53:49.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:49.929+0000 7f42ed400700 1 -- 172.21.15.27:0/1686467972 wait complete. 2024-04-16T05:53:50.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.089+0000 7f3152c00700 1 -- 172.21.15.27:0/2650741083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f314c0ff000 msgr2=0x7f314c0ff410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:50.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.089+0000 7f3152c00700 1 --2- 172.21.15.27:0/2650741083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f314c0ff000 0x7f314c0ff410 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f313c004940 tx=0x7f313c009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:50.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.089+0000 7f3152c00700 1 -- 172.21.15.27:0/2650741083 shutdown_connections 2024-04-16T05:53:50.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.089+0000 7f3152c00700 1 --2- 172.21.15.27:0/2650741083 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f314c1002b0 0x7f314c100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:50.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.089+0000 7f3152c00700 1 --2- 172.21.15.27:0/2650741083 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f314c0ff000 0x7f314c0ff410 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:50.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.089+0000 7f3152c00700 1 -- 172.21.15.27:0/2650741083 >> 172.21.15.27:0/2650741083 conn(0x7f314c0fa950 msgr2=0x7f314c0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:50.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.090+0000 7f3152c00700 1 -- 172.21.15.27:0/2650741083 shutdown_connections 2024-04-16T05:53:50.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.090+0000 7f3152c00700 1 -- 172.21.15.27:0/2650741083 wait complete. 2024-04-16T05:53:50.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.091+0000 7f3152c00700 1 Processor -- start 2024-04-16T05:53:50.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.091+0000 7f3152c00700 1 -- start start 2024-04-16T05:53:50.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.091+0000 7f3152c00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f314c0ff000 0x7f314c10fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:50.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.091+0000 7f3152c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f314c1002b0 0x7f314c110160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:50.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.091+0000 7f3152c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c110760 con 0x7f314c1002b0 2024-04-16T05:53:50.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.091+0000 7f3152c00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c110fb0 con 0x7f314c0ff000 2024-04-16T05:53:50.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.092+0000 7f314b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f314c1002b0 0x7f314c110160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:50.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.092+0000 7f314b400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f314c1002b0 0x7f314c110160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59638/0 (socket says 172.21.15.27:59638) 2024-04-16T05:53:50.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.092+0000 7f314b400700 1 -- 172.21.15.27:0/3836670795 learned_addr learned my addr 172.21.15.27:0/3836670795 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:53:50.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.092+0000 7f314be00700 1 --2- 172.21.15.27:0/3836670795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f314c0ff000 0x7f314c10fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:50.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.092+0000 7f314b400700 1 -- 172.21.15.27:0/3836670795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f314c0ff000 msgr2=0x7f314c10fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:50.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.092+0000 7f314b400700 1 --2- 172.21.15.27:0/3836670795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f314c0ff000 0x7f314c10fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:50.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.092+0000 7f314b400700 1 -- 172.21.15.27:0/3836670795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313c005040 con 0x7f314c1002b0 2024-04-16T05:53:50.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.093+0000 7f314b400700 1 --2- 172.21.15.27:0/3836670795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f314c1002b0 0x7f314c110160 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3138002a50 tx=0x7f31380046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:50.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.093+0000 7f3143e00700 1 -- 172.21.15.27:0/3836670795 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3138007cc0 con 0x7f314c1002b0 2024-04-16T05:53:50.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.093+0000 7f3143e00700 1 -- 172.21.15.27:0/3836670795 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f313800f040 con 0x7f314c1002b0 2024-04-16T05:53:50.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.093+0000 7f3152c00700 1 -- 172.21.15.27:0/3836670795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f314c111290 con 0x7f314c1002b0 2024-04-16T05:53:50.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.093+0000 7f3143e00700 1 -- 172.21.15.27:0/3836670795 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3138009640 con 0x7f314c1002b0 2024-04-16T05:53:50.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.093+0000 7f3152c00700 1 -- 172.21.15.27:0/3836670795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f314c1117d0 con 0x7f314c1002b0 2024-04-16T05:53:50.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.094+0000 7f3152c00700 1 -- 172.21.15.27:0/3836670795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f314c05f5f0 con 0x7f314c1002b0 2024-04-16T05:53:50.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.097+0000 7f3143e00700 1 -- 172.21.15.27:0/3836670795 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f313800d690 con 0x7f314c1002b0 2024-04-16T05:53:50.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.098+0000 7f3143e00700 1 --2- 172.21.15.27:0/3836670795 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f313006ec00 0x7f31300710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:50.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.098+0000 7f3143e00700 1 -- 172.21.15.27:0/3836670795 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f313808e860 con 0x7f314c1002b0 2024-04-16T05:53:50.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.099+0000 7f314be00700 1 --2- 172.21.15.27:0/3836670795 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f313006ec00 0x7f31300710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:50.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.099+0000 7f314be00700 1 --2- 172.21.15.27:0/3836670795 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f313006ec00 0x7f31300710b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f313c00b3b0 tx=0x7f313c009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:50.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.106+0000 7f3143e00700 1 -- 172.21.15.27:0/3836670795 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f313805af40 con 0x7f314c1002b0 2024-04-16T05:53:50.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.388+0000 7f3152c00700 1 -- 172.21.15.27:0/3836670795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f314c107af0 con 0x7f314c1002b0 2024-04-16T05:53:50.386 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.389+0000 7f3143e00700 1 -- 172.21.15.27:0/3836670795 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1936 (secure 0 0 0) 0x7f313805aad0 con 0x7f314c1002b0 2024-04-16T05:53:50.388 INFO:teuthology.orchestra.run.smithi027.stdout:e36 2024-04-16T05:53:50.388 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:53:50.388 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:53:50.388 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:53:50.389 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:53:50.389 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:53:50.389 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:53:50.389 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 35 2024-04-16T05:53:50.389 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:53:50.389 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:53:50.389 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:51:47.041625+0000 2024-04-16T05:53:50.389 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:53:50.389 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:53:50.389 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:53:50.389 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:53:50.389 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:53:50.390 INFO:teuthology.orchestra.run.smithi027.stdout:max_xattr_size 65536 2024-04-16T05:53:50.390 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:53:50.390 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:53:50.390 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 88 2024-04-16T05:53:50.390 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-16T05:53:50.390 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:53:50.390 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:53:50.390 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=34400} 2024-04-16T05:53:50.390 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:53:50.390 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:53:50.390 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:53:50.390 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:53:50.390 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:53:50.391 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:53:50.391 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:53:50.391 INFO:teuthology.orchestra.run.smithi027.stdout:bal_rank_mask -1 2024-04-16T05:53:50.391 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:53:50.391 INFO:teuthology.orchestra.run.smithi027.stdout:qdb_cluster leader: 34400 members: 34400 2024-04-16T05:53:50.391 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{0:34400} state up:active seq 23 join_fscid=1 addr [v2:172.21.15.27:6826/2831468256,v1:172.21.15.27:6827/2831468256] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:53:50.391 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:53:50.391 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:53:50.391 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:53:50.391 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:53:50.391 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{-1:34432} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6828/3219609959,v1:172.21.15.27:6829/3219609959] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:53:50.391 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:44337} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/97103507,v1:172.21.15.122:6825/97103507] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:53:50.391 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:44359} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/1030624816,v1:172.21.15.122:6827/1030624816] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:53:50.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.392+0000 7f3152c00700 1 -- 172.21.15.27:0/3836670795 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f313006ec00 msgr2=0x7f31300710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:50.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.392+0000 7f3152c00700 1 --2- 172.21.15.27:0/3836670795 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f313006ec00 0x7f31300710b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f313c00b3b0 tx=0x7f313c009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:50.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.392+0000 7f3152c00700 1 -- 172.21.15.27:0/3836670795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f314c1002b0 msgr2=0x7f314c110160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:50.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.392+0000 7f3152c00700 1 --2- 172.21.15.27:0/3836670795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f314c1002b0 0x7f314c110160 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3138002a50 tx=0x7f31380046b0 comp rx=0 tx=0).stop 2024-04-16T05:53:50.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.392+0000 7f3152c00700 1 -- 172.21.15.27:0/3836670795 shutdown_connections 2024-04-16T05:53:50.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.392+0000 7f3152c00700 1 --2- 172.21.15.27:0/3836670795 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f313006ec00 0x7f31300710b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:50.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.392+0000 7f3152c00700 1 --2- 172.21.15.27:0/3836670795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f314c0ff000 0x7f314c10fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:50.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.392+0000 7f3152c00700 1 --2- 172.21.15.27:0/3836670795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f314c1002b0 0x7f314c110160 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:50.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.392+0000 7f3152c00700 1 -- 172.21.15.27:0/3836670795 >> 172.21.15.27:0/3836670795 conn(0x7f314c0fa950 msgr2=0x7f314c0fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:50.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.392+0000 7f3152c00700 1 -- 172.21.15.27:0/3836670795 shutdown_connections 2024-04-16T05:53:50.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.392+0000 7f3152c00700 1 -- 172.21.15.27:0/3836670795 wait complete. 2024-04-16T05:53:50.393 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 36 2024-04-16T05:53:50.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.556+0000 7f1bab800700 1 -- 172.21.15.27:0/175837178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ba4071e50 msgr2=0x7f1ba4070350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:50.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.556+0000 7f1bab800700 1 --2- 172.21.15.27:0/175837178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ba4071e50 0x7f1ba4070350 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1b98004940 tx=0x7f1b98009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:50.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.556+0000 7f1bab800700 1 -- 172.21.15.27:0/175837178 shutdown_connections 2024-04-16T05:53:50.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.556+0000 7f1bab800700 1 --2- 172.21.15.27:0/175837178 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba4070890 0x7f1ba4070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:50.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.556+0000 7f1bab800700 1 --2- 172.21.15.27:0/175837178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ba4071e50 0x7f1ba4070350 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:50.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.556+0000 7f1bab800700 1 -- 172.21.15.27:0/175837178 >> 172.21.15.27:0/175837178 conn(0x7f1ba40f8540 msgr2=0x7f1ba40fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:50.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.556+0000 7f1bab800700 1 -- 172.21.15.27:0/175837178 shutdown_connections 2024-04-16T05:53:50.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.556+0000 7f1bab800700 1 -- 172.21.15.27:0/175837178 wait complete. 2024-04-16T05:53:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.557+0000 7f1bab800700 1 Processor -- start 2024-04-16T05:53:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.557+0000 7f1bab800700 1 -- start start 2024-04-16T05:53:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.557+0000 7f1bab800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ba4070890 0x7f1ba410fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.557+0000 7f1bab800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba4071e50 0x7f1ba41101a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.557+0000 7f1bab800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba4110830 con 0x7f1ba4070890 2024-04-16T05:53:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.557+0000 7f1bab800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba4111020 con 0x7f1ba4071e50 2024-04-16T05:53:50.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.558+0000 7f1ba8e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ba4070890 0x7f1ba410fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.558+0000 7f1ba8e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ba4070890 0x7f1ba410fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59654/0 (socket says 172.21.15.27:59654) 2024-04-16T05:53:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.558+0000 7f1ba8e00700 1 -- 172.21.15.27:0/211893362 learned_addr learned my addr 172.21.15.27:0/211893362 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:53:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.558+0000 7f1ba3e00700 1 --2- 172.21.15.27:0/211893362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba4071e50 0x7f1ba41101a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.558+0000 7f1ba8e00700 1 -- 172.21.15.27:0/211893362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba4071e50 msgr2=0x7f1ba41101a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.558+0000 7f1ba8e00700 1 --2- 172.21.15.27:0/211893362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba4071e50 0x7f1ba41101a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.558+0000 7f1ba8e00700 1 -- 172.21.15.27:0/211893362 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b98005040 con 0x7f1ba4070890 2024-04-16T05:53:50.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.559+0000 7f1ba8e00700 1 --2- 172.21.15.27:0/211893362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ba4070890 0x7f1ba410fc60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800b970 tx=0x7f1b98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:50.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.559+0000 7f1ba0c00700 1 -- 172.21.15.27:0/211893362 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b9800e040 con 0x7f1ba4070890 2024-04-16T05:53:50.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.559+0000 7f1ba0c00700 1 -- 172.21.15.27:0/211893362 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b9800bbf0 con 0x7f1ba4070890 2024-04-16T05:53:50.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.559+0000 7f1ba0c00700 1 -- 172.21.15.27:0/211893362 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b9801c630 con 0x7f1ba4070890 2024-04-16T05:53:50.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.559+0000 7f1bab800700 1 -- 172.21.15.27:0/211893362 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba41112a0 con 0x7f1ba4070890 2024-04-16T05:53:50.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.559+0000 7f1bab800700 1 -- 172.21.15.27:0/211893362 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba4111860 con 0x7f1ba4070890 2024-04-16T05:53:50.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.560+0000 7f1bab800700 1 -- 172.21.15.27:0/211893362 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba405f5f0 con 0x7f1ba4070890 2024-04-16T05:53:50.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.561+0000 7f1ba0c00700 1 -- 172.21.15.27:0/211893362 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1b9800bd60 con 0x7f1ba4070890 2024-04-16T05:53:50.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.562+0000 7f1ba0c00700 1 --2- 172.21.15.27:0/211893362 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1b8c06ec10 0x7f1b8c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:50.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.562+0000 7f1ba0c00700 1 -- 172.21.15.27:0/211893362 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f1b9808fe20 con 0x7f1ba4070890 2024-04-16T05:53:50.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.563+0000 7f1ba3e00700 1 --2- 172.21.15.27:0/211893362 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1b8c06ec10 0x7f1b8c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:50.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.563+0000 7f1ba3e00700 1 --2- 172.21.15.27:0/211893362 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1b8c06ec10 0x7f1b8c0710c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1b90006fd0 tx=0x7f1b90008040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:50.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.567+0000 7f1ba0c00700 1 -- 172.21.15.27:0/211893362 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1b9805c560 con 0x7f1ba4070890 2024-04-16T05:53:50.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:50 smithi122 ceph-mon[80930]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:53:50.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:50 smithi122 ceph-mon[80930]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:53:50.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:50 smithi122 ceph-mon[80930]: pgmap v292: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:50.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:50 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1686467972' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:50.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:50 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/3836670795' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:53:50.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:50 smithi027 ceph-mon[93635]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:53:50.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:50 smithi027 ceph-mon[93635]: from='client.34588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:53:50.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:50 smithi027 ceph-mon[93635]: pgmap v292: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:50.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:50 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1686467972' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:53:50.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:50 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/3836670795' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:53:50.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.818+0000 7f1bab800700 1 -- 172.21.15.27:0/211893362 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ba4111be0 con 0x7f1b8c06ec10 2024-04-16T05:53:50.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.821+0000 7f1ba0c00700 1 -- 172.21.15.27:0/211893362 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+435 (secure 0 0 0) 0x7f1ba4111be0 con 0x7f1b8c06ec10 2024-04-16T05:53:50.819 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:53:50.819 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa", 2024-04-16T05:53:50.819 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": true, 2024-04-16T05:53:50.819 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-16T05:53:50.819 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [ 2024-04-16T05:53:50.820 INFO:teuthology.orchestra.run.smithi027.stdout: "mds", 2024-04-16T05:53:50.820 INFO:teuthology.orchestra.run.smithi027.stdout: "mon", 2024-04-16T05:53:50.820 INFO:teuthology.orchestra.run.smithi027.stdout: "crash", 2024-04-16T05:53:50.820 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr", 2024-04-16T05:53:50.820 INFO:teuthology.orchestra.run.smithi027.stdout: "osd" 2024-04-16T05:53:50.820 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-04-16T05:53:50.820 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": "16/21 daemons upgraded", 2024-04-16T05:53:50.820 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "Currently upgrading grafana daemons", 2024-04-16T05:53:50.820 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:53:50.820 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:53:50.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.824+0000 7f1bab800700 1 -- 172.21.15.27:0/211893362 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1b8c06ec10 msgr2=0x7f1b8c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:50.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.824+0000 7f1bab800700 1 --2- 172.21.15.27:0/211893362 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1b8c06ec10 0x7f1b8c0710c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1b90006fd0 tx=0x7f1b90008040 comp rx=0 tx=0).stop 2024-04-16T05:53:50.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.824+0000 7f1bab800700 1 -- 172.21.15.27:0/211893362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ba4070890 msgr2=0x7f1ba410fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:50.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.824+0000 7f1bab800700 1 --2- 172.21.15.27:0/211893362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ba4070890 0x7f1ba410fc60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800b970 tx=0x7f1b98009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:50.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.824+0000 7f1bab800700 1 -- 172.21.15.27:0/211893362 shutdown_connections 2024-04-16T05:53:50.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.824+0000 7f1bab800700 1 --2- 172.21.15.27:0/211893362 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1b8c06ec10 0x7f1b8c0710c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:50.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.824+0000 7f1bab800700 1 --2- 172.21.15.27:0/211893362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ba4070890 0x7f1ba410fc60 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:50.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.824+0000 7f1bab800700 1 --2- 172.21.15.27:0/211893362 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba4071e50 0x7f1ba41101a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:50.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.824+0000 7f1bab800700 1 -- 172.21.15.27:0/211893362 >> 172.21.15.27:0/211893362 conn(0x7f1ba40f8540 msgr2=0x7f1ba40f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:50.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.824+0000 7f1bab800700 1 -- 172.21.15.27:0/211893362 shutdown_connections 2024-04-16T05:53:50.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.824+0000 7f1bab800700 1 -- 172.21.15.27:0/211893362 wait complete. 2024-04-16T05:53:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.982+0000 7ff8c3600700 1 -- 172.21.15.27:0/3898750591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8bc0662f0 msgr2=0x7ff8bc0ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.982+0000 7ff8c3600700 1 --2- 172.21.15.27:0/3898750591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8bc0662f0 0x7ff8bc0ffa30 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0004940 tx=0x7ff8b0009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.982+0000 7ff8c3600700 1 -- 172.21.15.27:0/3898750591 shutdown_connections 2024-04-16T05:53:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.982+0000 7ff8c3600700 1 --2- 172.21.15.27:0/3898750591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8bc0fff70 0x7ff8bc102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:50.980 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.982+0000 7ff8c3600700 1 --2- 172.21.15.27:0/3898750591 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8bc0662f0 0x7ff8bc0ffa30 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:50.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.982+0000 7ff8c3600700 1 -- 172.21.15.27:0/3898750591 >> 172.21.15.27:0/3898750591 conn(0x7ff8bc0f74c0 msgr2=0x7ff8bc0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:50.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.982+0000 7ff8c3600700 1 -- 172.21.15.27:0/3898750591 shutdown_connections 2024-04-16T05:53:50.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.982+0000 7ff8c3600700 1 -- 172.21.15.27:0/3898750591 wait complete. 2024-04-16T05:53:50.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.983+0000 7ff8c3600700 1 Processor -- start 2024-04-16T05:53:50.981 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.984+0000 7ff8c3600700 1 -- start start 2024-04-16T05:53:50.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.984+0000 7ff8c3600700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8bc0662f0 0x7ff8bc1070a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:50.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.984+0000 7ff8c3600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8bc0fff70 0x7ff8bc1075e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:50.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.984+0000 7ff8c3600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc107c30 con 0x7ff8bc0fff70 2024-04-16T05:53:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.984+0000 7ff8c3600700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc108480 con 0x7ff8bc0662f0 2024-04-16T05:53:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.984+0000 7ff8bbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8bc0fff70 0x7ff8bc1075e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.984+0000 7ff8bbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8bc0fff70 0x7ff8bc1075e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59676/0 (socket says 172.21.15.27:59676) 2024-04-16T05:53:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.984+0000 7ff8bbe00700 1 -- 172.21.15.27:0/293241863 learned_addr learned my addr 172.21.15.27:0/293241863 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:53:50.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.985+0000 7ff8c0c00700 1 --2- 172.21.15.27:0/293241863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8bc0662f0 0x7ff8bc1070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.985+0000 7ff8bbe00700 1 -- 172.21.15.27:0/293241863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8bc0662f0 msgr2=0x7ff8bc1070a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.985+0000 7ff8bbe00700 1 --2- 172.21.15.27:0/293241863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8bc0662f0 0x7ff8bc1070a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.985+0000 7ff8bbe00700 1 -- 172.21.15.27:0/293241863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b0005040 con 0x7ff8bc0fff70 2024-04-16T05:53:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.985+0000 7ff8c0c00700 1 --2- 172.21.15.27:0/293241863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8bc0662f0 0x7ff8bc1070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T05:53:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.985+0000 7ff8bbe00700 1 --2- 172.21.15.27:0/293241863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8bc0fff70 0x7ff8bc1075e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff8a800c8c0 tx=0x7ff8a80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.985+0000 7ff8b8c00700 1 -- 172.21.15.27:0/293241863 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8a800a980 con 0x7ff8bc0fff70 2024-04-16T05:53:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.985+0000 7ff8b8c00700 1 -- 172.21.15.27:0/293241863 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8a800ce60 con 0x7ff8bc0fff70 2024-04-16T05:53:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.986+0000 7ff8c3600700 1 -- 172.21.15.27:0/293241863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8bc108760 con 0x7ff8bc0fff70 2024-04-16T05:53:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.986+0000 7ff8b8c00700 1 -- 172.21.15.27:0/293241863 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8a80115e0 con 0x7ff8bc0fff70 2024-04-16T05:53:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.986+0000 7ff8c3600700 1 -- 172.21.15.27:0/293241863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8bc108d20 con 0x7ff8bc0fff70 2024-04-16T05:53:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.987+0000 7ff8c3600700 1 -- 172.21.15.27:0/293241863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8bc0f8ff0 con 0x7ff8bc0fff70 2024-04-16T05:53:50.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.989+0000 7ff8b8c00700 1 -- 172.21.15.27:0/293241863 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff8a8012400 con 0x7ff8bc0fff70 2024-04-16T05:53:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.990+0000 7ff8b8c00700 1 --2- 172.21.15.27:0/293241863 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff8a406ec10 0x7ff8a40710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:53:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.990+0000 7ff8b8c00700 1 -- 172.21.15.27:0/293241863 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7ff8a808eaf0 con 0x7ff8bc0fff70 2024-04-16T05:53:50.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.991+0000 7ff8c0c00700 1 --2- 172.21.15.27:0/293241863 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff8a406ec10 0x7ff8a40710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:53:50.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.991+0000 7ff8c0c00700 1 --2- 172.21.15.27:0/293241863 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff8a406ec10 0x7ff8a40710c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0000c00 tx=0x7ff8b0009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:53:50.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:50.995+0000 7ff8b8c00700 1 -- 172.21.15.27:0/293241863 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff8a8057740 con 0x7ff8bc0fff70 2024-04-16T05:53:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:51.376+0000 7ff8c3600700 1 -- 172.21.15.27:0/293241863 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7ff8bc05f5f0 con 0x7ff8bc0fff70 2024-04-16T05:53:51.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:51.377+0000 7ff8b8c00700 1 -- 172.21.15.27:0/293241863 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7ff8a80126b0 con 0x7ff8bc0fff70 2024-04-16T05:53:51.375 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:53:51.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:51.380+0000 7ff8c3600700 1 -- 172.21.15.27:0/293241863 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff8a406ec10 msgr2=0x7ff8a40710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:51.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:51.380+0000 7ff8c3600700 1 --2- 172.21.15.27:0/293241863 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff8a406ec10 0x7ff8a40710c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0000c00 tx=0x7ff8b0009e30 comp rx=0 tx=0).stop 2024-04-16T05:53:51.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:51.380+0000 7ff8c3600700 1 -- 172.21.15.27:0/293241863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8bc0fff70 msgr2=0x7ff8bc1075e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:53:51.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:51.380+0000 7ff8c3600700 1 --2- 172.21.15.27:0/293241863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8bc0fff70 0x7ff8bc1075e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff8a800c8c0 tx=0x7ff8a80044c0 comp rx=0 tx=0).stop 2024-04-16T05:53:51.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:51.380+0000 7ff8c3600700 1 -- 172.21.15.27:0/293241863 shutdown_connections 2024-04-16T05:53:51.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:51.380+0000 7ff8c3600700 1 --2- 172.21.15.27:0/293241863 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff8a406ec10 0x7ff8a40710c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:51.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:51.380+0000 7ff8c3600700 1 --2- 172.21.15.27:0/293241863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8bc0662f0 0x7ff8bc1070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:51.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:51.380+0000 7ff8c3600700 1 --2- 172.21.15.27:0/293241863 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff8bc0fff70 0x7ff8bc1075e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:53:51.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:51.380+0000 7ff8c3600700 1 -- 172.21.15.27:0/293241863 >> 172.21.15.27:0/293241863 conn(0x7ff8bc0f74c0 msgr2=0x7ff8bc0fbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:53:51.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:51.380+0000 7ff8c3600700 1 -- 172.21.15.27:0/293241863 shutdown_connections 2024-04-16T05:53:51.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:53:51.380+0000 7ff8c3600700 1 -- 172.21.15.27:0/293241863 wait complete. 2024-04-16T05:53:51.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:51 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/293241863' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:53:51.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:51 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:51.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:51 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:51.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:51 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/293241863' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:53:51.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:51 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:51.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:51 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:52.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:52 smithi122 ceph-mon[80930]: from='client.34600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:53:52.772 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:52 smithi122 ceph-mon[80930]: pgmap v293: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:52.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:52 smithi027 ceph-mon[93635]: from='client.34600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:53:52.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:52 smithi027 ceph-mon[93635]: pgmap v293: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:54.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:54 smithi122 ceph-mon[80930]: pgmap v294: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:54.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:54 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:54.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:54 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:54.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:54 smithi027 ceph-mon[93635]: pgmap v294: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:54.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:54 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:54.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:54 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:53:56.766 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:56 smithi027 ceph-mon[93635]: pgmap v295: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:56.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:56 smithi122 ceph-mon[80930]: pgmap v295: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:58.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 646, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 142942 2024-04-16T05:53:58.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 646, success, outsize: 120 2024-04-16T05:53:58.041 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 648, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143029 2024-04-16T05:53:58.042 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 648, success, outsize: 120 2024-04-16T05:53:58.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 650, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143119 2024-04-16T05:53:58.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 650, success, outsize: 120 2024-04-16T05:53:58.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 652, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143215 2024-04-16T05:53:58.095 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 652, success, outsize: 120 2024-04-16T05:53:58.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 654, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143308 2024-04-16T05:53:58.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 654, success, outsize: 120 2024-04-16T05:53:58.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 656, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143395 2024-04-16T05:53:58.158 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 656, success, outsize: 120 2024-04-16T05:53:58.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 658, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143485 2024-04-16T05:53:58.191 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 658, success, outsize: 120 2024-04-16T05:53:58.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 660, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143581 2024-04-16T05:53:58.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 660, success, outsize: 120 2024-04-16T05:53:58.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 662, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143674 2024-04-16T05:53:58.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 662, success, outsize: 120 2024-04-16T05:53:58.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 664, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143761 2024-04-16T05:53:58.290 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 664, success, outsize: 120 2024-04-16T05:53:58.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 666, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143851 2024-04-16T05:53:58.314 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 666, success, outsize: 120 2024-04-16T05:53:58.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 668, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 143947 2024-04-16T05:53:58.346 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 668, success, outsize: 120 2024-04-16T05:53:58.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 670, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 144040 2024-04-16T05:53:58.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 670, success, outsize: 120 2024-04-16T05:53:58.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 672, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 144127 2024-04-16T05:53:58.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 672, success, outsize: 120 2024-04-16T05:53:58.439 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 674, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 144217 2024-04-16T05:53:58.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 674, success, outsize: 120 2024-04-16T05:53:58.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 676, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 144313 2024-04-16T05:53:58.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 676, success, outsize: 120 2024-04-16T05:53:58.499 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 678, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 144406 2024-04-16T05:53:58.500 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 678, success, outsize: 120 2024-04-16T05:53:58.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 680, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 144493 2024-04-16T05:53:58.529 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 680, success, outsize: 120 2024-04-16T05:53:58.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 682, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 144583 2024-04-16T05:53:58.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 682, success, outsize: 120 2024-04-16T05:53:58.591 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 684, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 144679 2024-04-16T05:53:58.591 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 684, success, outsize: 120 2024-04-16T05:53:58.621 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 686, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 144772 2024-04-16T05:53:58.621 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 686, success, outsize: 120 2024-04-16T05:53:58.650 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 688, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 144859 2024-04-16T05:53:58.651 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 688, success, outsize: 120 2024-04-16T05:53:58.675 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 690, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 144949 2024-04-16T05:53:58.675 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 690, success, outsize: 120 2024-04-16T05:53:58.702 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 692, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 145045 2024-04-16T05:53:58.703 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 692, success, outsize: 120 2024-04-16T05:53:58.733 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 694, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 145138 2024-04-16T05:53:58.733 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 694, success, outsize: 120 2024-04-16T05:53:58.759 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 696, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 145225 2024-04-16T05:53:58.759 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 696, success, outsize: 120 2024-04-16T05:53:58.787 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 698, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 145315 2024-04-16T05:53:58.788 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 698, success, outsize: 120 2024-04-16T05:53:58.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:58 smithi027 ceph-mon[93635]: pgmap v296: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:58.820 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:unique: 700, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 145411 2024-04-16T05:53:58.820 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: unique: 700, success, outsize: 120 2024-04-16T05:53:59.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:58 smithi122 ceph-mon[80930]: pgmap v296: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:53:59.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:53:59 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:54:00.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:53:59 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:54:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: pgmap v297: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:54:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:54:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:54:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:54:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:54:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T05:54:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:54:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:00.808 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:00.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:00.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:00.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:00.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:00.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:00.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:00.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:00.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:00.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:00.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:00.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:00.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:00.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:00.810 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:54:00.810 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-16T05:54:00.810 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-16T05:54:00.810 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T05:54:00.810 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-16T05:54:00.810 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-16T05:54:00.810 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-16T05:54:00.810 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-16T05:54:00.811 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-16T05:54:00.811 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-16T05:54:00.814 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-16T05:54:00.814 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-16T05:54:00.814 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-16T05:54:00.814 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-16T05:54:00.814 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-16T05:54:00.814 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T05:54:00.814 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-16T05:54:00.814 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-16T05:54:00.814 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-16T05:54:00.814 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-16T05:54:00.814 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-16T05:54:00.814 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-16T05:54:00.815 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-16T05:54:00.815 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-16T05:54:00.815 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T05:54:00.815 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T05:54:00.815 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T05:54:00.815 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T05:54:00.815 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T05:54:00.815 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T05:54:00.815 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-16T05:54:00.815 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-16T05:54:00.816 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:54:00.816 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:54:00.816 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:54:00.816 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:00 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:54:01.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: pgmap v297: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:01.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:54:01.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:54:01.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:54:01.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:54:01.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:54:01.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T05:54:01.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:54:01.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:01.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:01.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:01.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:01.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:01.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:01.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:01.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:01.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:01.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:01.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:01.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:01.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:01.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:01.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:01.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:01.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:01.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:01.024 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:54:01.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-04-16T05:54:01.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-04-16T05:54:01.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T05:54:01.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-04-16T05:54:01.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-04-16T05:54:01.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-04-16T05:54:01.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-04-16T05:54:01.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-04-16T05:54:01.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-04-16T05:54:01.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-04-16T05:54:01.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-04-16T05:54:01.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-04-16T05:54:01.025 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-04-16T05:54:01.026 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-04-16T05:54:01.026 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T05:54:01.026 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-04-16T05:54:01.026 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-04-16T05:54:01.026 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-04-16T05:54:01.026 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-04-16T05:54:01.026 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-04-16T05:54:01.026 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-04-16T05:54:01.026 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-04-16T05:54:01.026 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-04-16T05:54:01.026 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T05:54:01.026 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T05:54:01.026 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T05:54:01.026 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T05:54:01.027 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T05:54:01.027 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-04-16T05:54:01.027 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-04-16T05:54:01.027 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-04-16T05:54:01.027 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:54:01.027 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:54:01.027 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:54:01.027 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:00 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:54:01.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:01 smithi027 ceph-mon[93635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T05:54:01.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:01 smithi027 ceph-mon[93635]: Upgrade: Finalizing container_image settings 2024-04-16T05:54:01.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:01 smithi027 ceph-mon[93635]: Upgrade: Complete! 2024-04-16T05:54:01.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:01 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:54:01.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:01 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:54:01.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:01 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:54:01.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:01 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:54:02.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:01 smithi122 ceph-mon[80930]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-16T05:54:02.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:01 smithi122 ceph-mon[80930]: Upgrade: Finalizing container_image settings 2024-04-16T05:54:02.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:01 smithi122 ceph-mon[80930]: Upgrade: Complete! 2024-04-16T05:54:02.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:01 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:54:02.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:01 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:54:02.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:01 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:54:02.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:01 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:54:02.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:02 smithi027 ceph-mon[93635]: pgmap v298: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:03.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:02 smithi122 ceph-mon[80930]: pgmap v298: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:04.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:04 smithi027 ceph-mon[93635]: pgmap v299: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:04.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:04 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:54:05.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:04 smithi122 ceph-mon[80930]: pgmap v299: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:05.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:04 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:54:07.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:06 smithi122 ceph-mon[80930]: pgmap v300: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:07.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:06 smithi027 ceph-mon[93635]: pgmap v300: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:08.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:08 smithi027 ceph-mon[93635]: pgmap v301: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:09.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:08 smithi122 ceph-mon[80930]: pgmap v301: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:11.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:10 smithi122 ceph-mon[80930]: pgmap v302: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:11.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:10 smithi027 ceph-mon[93635]: pgmap v302: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:12.859 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:12 smithi027 ceph-mon[93635]: pgmap v303: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:13.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:12 smithi122 ceph-mon[80930]: pgmap v303: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:14 smithi122 ceph-mon[80930]: pgmap v304: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:54:15.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:14 smithi027 ceph-mon[93635]: pgmap v304: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:15.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:54:17.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:16 smithi122 ceph-mon[80930]: pgmap v305: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:17.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:16 smithi027 ceph-mon[93635]: pgmap v305: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:18.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:18 smithi027 ceph-mon[93635]: pgmap v306: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:19.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:18 smithi122 ceph-mon[80930]: pgmap v306: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:21.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:20 smithi122 ceph-mon[80930]: pgmap v307: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:21.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:20 smithi027 ceph-mon[93635]: pgmap v307: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:21.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.558+0000 7f286d800700 1 -- 172.21.15.27:0/3581823401 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28680ff000 msgr2=0x7f28680ff410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:54:21.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.558+0000 7f286d800700 1 --2- 172.21.15.27:0/3581823401 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28680ff000 0x7f28680ff410 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2858004940 tx=0x7f2858009e30 comp rx=0 tx=0).stop 2024-04-16T05:54:21.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.558+0000 7f286d800700 1 -- 172.21.15.27:0/3581823401 shutdown_connections 2024-04-16T05:54:21.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.558+0000 7f286d800700 1 --2- 172.21.15.27:0/3581823401 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28681002b0 0x7f2868100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:21.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.558+0000 7f286d800700 1 --2- 172.21.15.27:0/3581823401 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28680ff000 0x7f28680ff410 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:21.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.558+0000 7f286d800700 1 -- 172.21.15.27:0/3581823401 >> 172.21.15.27:0/3581823401 conn(0x7f28680fa950 msgr2=0x7f28680fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:54:21.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.558+0000 7f286d800700 1 -- 172.21.15.27:0/3581823401 shutdown_connections 2024-04-16T05:54:21.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.558+0000 7f286d800700 1 -- 172.21.15.27:0/3581823401 wait complete. 2024-04-16T05:54:21.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.560+0000 7f286d800700 1 Processor -- start 2024-04-16T05:54:21.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.560+0000 7f286d800700 1 -- start start 2024-04-16T05:54:21.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.560+0000 7f286d800700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28680ff000 0x7f286810fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:54:21.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.560+0000 7f286d800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28681002b0 0x7f2868110160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:54:21.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.560+0000 7f286d800700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2868110760 con 0x7f28681002b0 2024-04-16T05:54:21.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.560+0000 7f286d800700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2868110fb0 con 0x7f28680ff000 2024-04-16T05:54:21.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.561+0000 7f285fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28681002b0 0x7f2868110160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:54:21.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.561+0000 7f285fe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28681002b0 0x7f2868110160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41190/0 (socket says 172.21.15.27:41190) 2024-04-16T05:54:21.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.561+0000 7f285fe00700 1 -- 172.21.15.27:0/3488017826 learned_addr learned my addr 172.21.15.27:0/3488017826 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:54:21.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.561+0000 7f2866a00700 1 --2- 172.21.15.27:0/3488017826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28680ff000 0x7f286810fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:54:21.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.561+0000 7f285fe00700 1 -- 172.21.15.27:0/3488017826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28680ff000 msgr2=0x7f286810fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:54:21.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.561+0000 7f285fe00700 1 --2- 172.21.15.27:0/3488017826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28680ff000 0x7f286810fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:21.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.562+0000 7f285fe00700 1 -- 172.21.15.27:0/3488017826 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2858005040 con 0x7f28681002b0 2024-04-16T05:54:21.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.562+0000 7f2866a00700 1 --2- 172.21.15.27:0/3488017826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28680ff000 0x7f286810fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T05:54:21.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.562+0000 7f285fe00700 1 --2- 172.21.15.27:0/3488017826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28681002b0 0x7f2868110160 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2850002a50 tx=0x7f28500046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:54:21.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.562+0000 7f285ea00700 1 -- 172.21.15.27:0/3488017826 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2850007cc0 con 0x7f28681002b0 2024-04-16T05:54:21.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.562+0000 7f285ea00700 1 -- 172.21.15.27:0/3488017826 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f285000f040 con 0x7f28681002b0 2024-04-16T05:54:21.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.562+0000 7f286d800700 1 -- 172.21.15.27:0/3488017826 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2868111290 con 0x7f28681002b0 2024-04-16T05:54:21.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.562+0000 7f285ea00700 1 -- 172.21.15.27:0/3488017826 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2850009640 con 0x7f28681002b0 2024-04-16T05:54:21.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.563+0000 7f286d800700 1 -- 172.21.15.27:0/3488017826 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28681117d0 con 0x7f28681002b0 2024-04-16T05:54:21.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.564+0000 7f286d800700 1 -- 172.21.15.27:0/3488017826 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286805f5f0 con 0x7f28681002b0 2024-04-16T05:54:21.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.565+0000 7f285ea00700 1 -- 172.21.15.27:0/3488017826 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f28500097c0 con 0x7f28681002b0 2024-04-16T05:54:21.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.566+0000 7f285ea00700 1 --2- 172.21.15.27:0/3488017826 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f284c06ec10 0x7f284c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:54:21.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.566+0000 7f285ea00700 1 -- 172.21.15.27:0/3488017826 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f285008e8c0 con 0x7f28681002b0 2024-04-16T05:54:21.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.567+0000 7f2866a00700 1 --2- 172.21.15.27:0/3488017826 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f284c06ec10 0x7f284c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:54:21.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.567+0000 7f2866a00700 1 --2- 172.21.15.27:0/3488017826 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f284c06ec10 0x7f284c0710c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f285800b020 tx=0x7f2858009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:54:21.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.571+0000 7f285ea00700 1 -- 172.21.15.27:0/3488017826 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f285005af00 con 0x7f28681002b0 2024-04-16T05:54:21.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.824+0000 7f286d800700 1 -- 172.21.15.27:0/3488017826 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f2868111bc0 con 0x7f284c06ec10 2024-04-16T05:54:21.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.827+0000 7f285ea00700 1 -- 172.21.15.27:0/3488017826 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7f2868111bc0 con 0x7f284c06ec10 2024-04-16T05:54:21.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.830+0000 7f286d800700 1 -- 172.21.15.27:0/3488017826 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f284c06ec10 msgr2=0x7f284c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:54:21.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.830+0000 7f286d800700 1 --2- 172.21.15.27:0/3488017826 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f284c06ec10 0x7f284c0710c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f285800b020 tx=0x7f2858009e30 comp rx=0 tx=0).stop 2024-04-16T05:54:21.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.830+0000 7f286d800700 1 -- 172.21.15.27:0/3488017826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28681002b0 msgr2=0x7f2868110160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:54:21.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.830+0000 7f286d800700 1 --2- 172.21.15.27:0/3488017826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28681002b0 0x7f2868110160 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2850002a50 tx=0x7f28500046b0 comp rx=0 tx=0).stop 2024-04-16T05:54:21.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.830+0000 7f286d800700 1 -- 172.21.15.27:0/3488017826 shutdown_connections 2024-04-16T05:54:21.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.830+0000 7f286d800700 1 --2- 172.21.15.27:0/3488017826 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f284c06ec10 0x7f284c0710c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:21.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.830+0000 7f286d800700 1 --2- 172.21.15.27:0/3488017826 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f28680ff000 0x7f286810fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:21.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.830+0000 7f286d800700 1 --2- 172.21.15.27:0/3488017826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28681002b0 0x7f2868110160 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:21.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.830+0000 7f286d800700 1 -- 172.21.15.27:0/3488017826 >> 172.21.15.27:0/3488017826 conn(0x7f28680fa950 msgr2=0x7f28680fd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:54:21.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.831+0000 7f286d800700 1 -- 172.21.15.27:0/3488017826 shutdown_connections 2024-04-16T05:54:21.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:21.831+0000 7f286d800700 1 -- 172.21.15.27:0/3488017826 wait complete. 2024-04-16T05:54:22.618 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph orch ps' 2024-04-16T05:54:22.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:22 smithi027 ceph-mon[93635]: pgmap v308: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:23.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:22 smithi122 ceph-mon[80930]: pgmap v308: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:23.189 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:54:23.876 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:23 smithi027 ceph-mon[93635]: from='client.34608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:54:24.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:23 smithi122 ceph-mon[80930]: from='client.34608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:54:24.722 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:24 smithi027 ceph-mon[93635]: pgmap v309: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:24.923 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:24 smithi122 ceph-mon[80930]: pgmap v309: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.732+0000 7f1bc2c00700 1 -- 172.21.15.27:0/745006488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bbc070300 msgr2=0x7f1bbc0706d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:54:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.732+0000 7f1bc2c00700 1 --2- 172.21.15.27:0/745006488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bbc070300 0x7f1bbc0706d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1bac009d10 tx=0x7f1bac004d10 comp rx=0 tx=0).stop 2024-04-16T05:54:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.732+0000 7f1bc2c00700 1 -- 172.21.15.27:0/745006488 shutdown_connections 2024-04-16T05:54:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.732+0000 7f1bc2c00700 1 --2- 172.21.15.27:0/745006488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bbc070c10 0x7f1bbc109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.732+0000 7f1bc2c00700 1 --2- 172.21.15.27:0/745006488 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bbc070300 0x7f1bbc0706d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.732+0000 7f1bc2c00700 1 -- 172.21.15.27:0/745006488 >> 172.21.15.27:0/745006488 conn(0x7f1bbc0f89b0 msgr2=0x7f1bbc0fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:54:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.733+0000 7f1bc2c00700 1 -- 172.21.15.27:0/745006488 shutdown_connections 2024-04-16T05:54:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.733+0000 7f1bc2c00700 1 -- 172.21.15.27:0/745006488 wait complete. 2024-04-16T05:54:25.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.733+0000 7f1bc2c00700 1 Processor -- start 2024-04-16T05:54:25.732 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.734+0000 7f1bc2c00700 1 -- start start 2024-04-16T05:54:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.734+0000 7f1bc2c00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bbc070300 0x7f1bbc115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:54:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.734+0000 7f1bc2c00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bbc070c10 0x7f1bbc116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:54:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.734+0000 7f1bc2c00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bbc1167b0 con 0x7f1bbc070300 2024-04-16T05:54:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.734+0000 7f1bc2c00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bbc10fd90 con 0x7f1bbc070c10 2024-04-16T05:54:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.734+0000 7f1bbbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bbc070300 0x7f1bbc115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:54:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.734+0000 7f1bbbe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bbc070300 0x7f1bbc115c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35112/0 (socket says 172.21.15.27:35112) 2024-04-16T05:54:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.734+0000 7f1bbbe00700 1 -- 172.21.15.27:0/3917344429 learned_addr learned my addr 172.21.15.27:0/3917344429 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:54:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.735+0000 7f1bbb400700 1 --2- 172.21.15.27:0/3917344429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bbc070c10 0x7f1bbc116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:54:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.735+0000 7f1bbbe00700 1 -- 172.21.15.27:0/3917344429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bbc070c10 msgr2=0x7f1bbc116160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:54:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.735+0000 7f1bbbe00700 1 --2- 172.21.15.27:0/3917344429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bbc070c10 0x7f1bbc116160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.735+0000 7f1bbbe00700 1 -- 172.21.15.27:0/3917344429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bac009940 con 0x7f1bbc070300 2024-04-16T05:54:25.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.735+0000 7f1bbb400700 1 --2- 172.21.15.27:0/3917344429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bbc070c10 0x7f1bbc116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:54:25.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.735+0000 7f1bbbe00700 1 --2- 172.21.15.27:0/3917344429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bbc070300 0x7f1bbc115c20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1bac008df0 tx=0x7f1bac004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:54:25.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.736+0000 7f1bb3e00700 1 -- 172.21.15.27:0/3917344429 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bac00e040 con 0x7f1bbc070300 2024-04-16T05:54:25.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.736+0000 7f1bb3e00700 1 -- 172.21.15.27:0/3917344429 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1bac0049f0 con 0x7f1bbc070300 2024-04-16T05:54:25.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.736+0000 7f1bb3e00700 1 -- 172.21.15.27:0/3917344429 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bac01c640 con 0x7f1bbc070300 2024-04-16T05:54:25.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.736+0000 7f1bc2c00700 1 -- 172.21.15.27:0/3917344429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bbc110010 con 0x7f1bbc070300 2024-04-16T05:54:25.734 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.736+0000 7f1bc2c00700 1 -- 172.21.15.27:0/3917344429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bbc110550 con 0x7f1bbc070300 2024-04-16T05:54:25.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.737+0000 7f1bc2c00700 1 -- 172.21.15.27:0/3917344429 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bbc071a40 con 0x7f1bbc070300 2024-04-16T05:54:25.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.738+0000 7f1bb3e00700 1 -- 172.21.15.27:0/3917344429 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f1bac00bd00 con 0x7f1bbc070300 2024-04-16T05:54:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.739+0000 7f1bb3e00700 1 --2- 172.21.15.27:0/3917344429 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1ba006ec00 0x7f1ba00710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:54:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.739+0000 7f1bb3e00700 1 -- 172.21.15.27:0/3917344429 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f1bac0904b0 con 0x7f1bbc070300 2024-04-16T05:54:25.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.739+0000 7f1bbb400700 1 --2- 172.21.15.27:0/3917344429 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1ba006ec00 0x7f1ba00710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:54:25.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.740+0000 7f1bbb400700 1 --2- 172.21.15.27:0/3917344429 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1ba006ec00 0x7f1ba00710b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8004600 tx=0x7f1ba800b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:54:25.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.745+0000 7f1bb3e00700 1 -- 172.21.15.27:0/3917344429 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1bac05cc10 con 0x7f1bbc070300 2024-04-16T05:54:25.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:25.989+0000 7f1bc2c00700 1 -- 172.21.15.27:0/3917344429 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1bbc111270 con 0x7f1ba006ec00 2024-04-16T05:54:26.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:26.015+0000 7f1bb3e00700 1 -- 172.21.15.27:0/3917344429 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f1bbc111270 con 0x7f1ba006ec00 2024-04-16T05:54:26.013 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:54:26.014 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (78s) 34s ago 21m 13.7M - 0.25.0 c8568f914cd2 42321467ded6 2024-04-16T05:54:26.014 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (7m) 34s ago 21m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 4aa1f0006685 2024-04-16T05:54:26.014 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (7m) 2m ago 20m 6656k - 19.0.0-2884-g218a6914 a4e4811815e4 1c1912507d56 2024-04-16T05:54:26.014 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (43s) 34s ago 21m 53.0M - 9.4.12 2bacad6d85d8 ec2f3705610a 2024-04-16T05:54:26.014 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (3m) 34s ago 14m 12.1M - 19.0.0-2884-g218a6914 a4e4811815e4 b3c2178c537e 2024-04-16T05:54:26.014 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (4m) 34s ago 14m 21.6M - 19.0.0-2884-g218a6914 a4e4811815e4 14b0c83f2891 2024-04-16T05:54:26.014 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (3m) 2m ago 14m 11.2M - 19.0.0-2884-g218a6914 a4e4811815e4 8ba95adfaaef 2024-04-16T05:54:26.014 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (2m) 2m ago 13m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 c60e9ca0f1c8 2024-04-16T05:54:26.014 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:8443,9283,8765 running (10m) 34s ago 23m 521M - 19.0.0-2884-g218a6914 a4e4811815e4 b69a0b08e12b 2024-04-16T05:54:26.014 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283,8765 running (9m) 2m ago 20m 454M - 19.0.0-2884-g218a6914 a4e4811815e4 32e27bcd7e42 2024-04-16T05:54:26.014 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (9m) 34s ago 23m 55.6M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 299ca337e4ca 2024-04-16T05:54:26.014 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (8m) 2m ago 20m 40.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5db5f3037ed0 2024-04-16T05:54:26.014 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (2m) 34s ago 21m 7591k - 1.5.0 0da6a335fe13 f3382c6beacb 2024-04-16T05:54:26.015 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (2m) 2m ago 20m 3078k - 1.5.0 0da6a335fe13 1f6817821f3a 2024-04-16T05:54:26.015 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (6m) 34s ago 19m 47.4M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 cef03b67bc5a 2024-04-16T05:54:26.015 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (6m) 34s ago 18m 46.3M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3a6ab1fc6e79 2024-04-16T05:54:26.015 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (5m) 34s ago 17m 44.9M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3524731d7277 2024-04-16T05:54:26.015 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (5m) 2m ago 17m 44.7M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 897d3579f929 2024-04-16T05:54:26.015 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (4m) 2m ago 16m 42.2M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 fa97758ff7c5 2024-04-16T05:54:26.015 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (4m) 2m ago 16m 49.6M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 fd12260a820d 2024-04-16T05:54:26.015 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (104s) 34s ago 20m 51.2M - 2.43.0 a07b618ecd1d a6cddf5443d5 2024-04-16T05:54:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:26.018+0000 7f1bc2c00700 1 -- 172.21.15.27:0/3917344429 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1ba006ec00 msgr2=0x7f1ba00710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:54:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:26.018+0000 7f1bc2c00700 1 --2- 172.21.15.27:0/3917344429 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1ba006ec00 0x7f1ba00710b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1ba8004600 tx=0x7f1ba800b040 comp rx=0 tx=0).stop 2024-04-16T05:54:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:26.018+0000 7f1bc2c00700 1 -- 172.21.15.27:0/3917344429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bbc070300 msgr2=0x7f1bbc115c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:54:26.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:26.018+0000 7f1bc2c00700 1 --2- 172.21.15.27:0/3917344429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bbc070300 0x7f1bbc115c20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f1bac008df0 tx=0x7f1bac004d10 comp rx=0 tx=0).stop 2024-04-16T05:54:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:26.018+0000 7f1bc2c00700 1 -- 172.21.15.27:0/3917344429 shutdown_connections 2024-04-16T05:54:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:26.018+0000 7f1bc2c00700 1 --2- 172.21.15.27:0/3917344429 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f1ba006ec00 0x7f1ba00710b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:26.018+0000 7f1bc2c00700 1 --2- 172.21.15.27:0/3917344429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1bbc070300 0x7f1bbc115c20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:26.018+0000 7f1bc2c00700 1 --2- 172.21.15.27:0/3917344429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1bbc070c10 0x7f1bbc116160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:26.018+0000 7f1bc2c00700 1 -- 172.21.15.27:0/3917344429 >> 172.21.15.27:0/3917344429 conn(0x7f1bbc0f89b0 msgr2=0x7f1bbc0f9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:54:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:26.019+0000 7f1bc2c00700 1 -- 172.21.15.27:0/3917344429 shutdown_connections 2024-04-16T05:54:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:26.019+0000 7f1bc2c00700 1 -- 172.21.15.27:0/3917344429 wait complete. 2024-04-16T05:54:27.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:26 smithi122 ceph-mon[80930]: pgmap v310: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:27.037 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:26 smithi027 ceph-mon[93635]: pgmap v310: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:27.118 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph orch upgrade status' 2024-04-16T05:54:27.666 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:54:28.001 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:27 smithi027 ceph-mon[93635]: from='client.34612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:54:28.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:27 smithi122 ceph-mon[80930]: from='client.34612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:54:28.859 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:28 smithi027 ceph-mon[93635]: pgmap v311: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:29.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:28 smithi122 ceph-mon[80930]: pgmap v311: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:29.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.874+0000 7fa4ae400700 1 -- 172.21.15.27:0/2715027026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4a8105830 msgr2=0x7fa4a80feef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:54:29.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.874+0000 7fa4ae400700 1 --2- 172.21.15.27:0/2715027026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4a8105830 0x7fa4a80feef0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa494004940 tx=0x7fa494009e30 comp rx=0 tx=0).stop 2024-04-16T05:54:29.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.874+0000 7fa4ae400700 1 -- 172.21.15.27:0/2715027026 shutdown_connections 2024-04-16T05:54:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.874+0000 7fa4ae400700 1 --2- 172.21.15.27:0/2715027026 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4a8105830 0x7fa4a80feef0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.874+0000 7fa4ae400700 1 --2- 172.21.15.27:0/2715027026 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4a8104e90 0x7fa4a8105260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.874+0000 7fa4ae400700 1 -- 172.21.15.27:0/2715027026 >> 172.21.15.27:0/2715027026 conn(0x7fa4a80fab10 msgr2=0x7fa4a80fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:54:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.874+0000 7fa4ae400700 1 -- 172.21.15.27:0/2715027026 shutdown_connections 2024-04-16T05:54:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.874+0000 7fa4ae400700 1 -- 172.21.15.27:0/2715027026 wait complete. 2024-04-16T05:54:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.875+0000 7fa4ae400700 1 Processor -- start 2024-04-16T05:54:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.875+0000 7fa4ae400700 1 -- start start 2024-04-16T05:54:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.876+0000 7fa4ae400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4a8104e90 0x7fa4a8115b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:54:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.876+0000 7fa4ae400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4a8105830 0x7fa4a81160c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:54:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.876+0000 7fa4ae400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a8116710 con 0x7fa4a8105830 2024-04-16T05:54:29.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.876+0000 7fa4ae400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a810fb20 con 0x7fa4a8104e90 2024-04-16T05:54:29.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.876+0000 7fa4a7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4a8105830 0x7fa4a81160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:54:29.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.876+0000 7fa4a7e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4a8105830 0x7fa4a81160c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35132/0 (socket says 172.21.15.27:35132) 2024-04-16T05:54:29.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.876+0000 7fa4a7e00700 1 -- 172.21.15.27:0/3134715381 learned_addr learned my addr 172.21.15.27:0/3134715381 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:54:29.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.876+0000 7fa4ad000700 1 --2- 172.21.15.27:0/3134715381 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4a8104e90 0x7fa4a8115b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:54:29.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.876+0000 7fa4a7e00700 1 -- 172.21.15.27:0/3134715381 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4a8104e90 msgr2=0x7fa4a8115b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:54:29.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.876+0000 7fa4a7e00700 1 --2- 172.21.15.27:0/3134715381 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4a8104e90 0x7fa4a8115b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:29.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.876+0000 7fa4a7e00700 1 -- 172.21.15.27:0/3134715381 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa494005040 con 0x7fa4a8105830 2024-04-16T05:54:29.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.876+0000 7fa4ad000700 1 --2- 172.21.15.27:0/3134715381 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4a8104e90 0x7fa4a8115b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:54:29.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.877+0000 7fa4a7e00700 1 --2- 172.21.15.27:0/3134715381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4a8105830 0x7fa4a81160c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa494009100 tx=0x7fa494009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:54:29.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.877+0000 7fa4a4c00700 1 -- 172.21.15.27:0/3134715381 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa49400e040 con 0x7fa4a8105830 2024-04-16T05:54:29.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.877+0000 7fa4a4c00700 1 -- 172.21.15.27:0/3134715381 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa494013420 con 0x7fa4a8105830 2024-04-16T05:54:29.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.877+0000 7fa4ae400700 1 -- 172.21.15.27:0/3134715381 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a810fda0 con 0x7fa4a8105830 2024-04-16T05:54:29.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.877+0000 7fa4a4c00700 1 -- 172.21.15.27:0/3134715381 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa49401c630 con 0x7fa4a8105830 2024-04-16T05:54:29.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.877+0000 7fa4ae400700 1 -- 172.21.15.27:0/3134715381 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a8110360 con 0x7fa4a8105830 2024-04-16T05:54:29.877 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.879+0000 7fa4ae400700 1 -- 172.21.15.27:0/3134715381 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a81005e0 con 0x7fa4a8105830 2024-04-16T05:54:29.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.880+0000 7fa4a4c00700 1 -- 172.21.15.27:0/3134715381 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa494013590 con 0x7fa4a8105830 2024-04-16T05:54:29.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.881+0000 7fa4a4c00700 1 --2- 172.21.15.27:0/3134715381 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa49006ec00 0x7fa4900710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:54:29.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.881+0000 7fa4a4c00700 1 -- 172.21.15.27:0/3134715381 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7fa49408fd50 con 0x7fa4a8105830 2024-04-16T05:54:29.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.881+0000 7fa4ad000700 1 --2- 172.21.15.27:0/3134715381 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa49006ec00 0x7fa4900710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:54:29.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.882+0000 7fa4ad000700 1 --2- 172.21.15.27:0/3134715381 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa49006ec00 0x7fa4900710b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa49c007fd0 tx=0x7fa49c009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:54:29.884 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:29.886+0000 7fa4a4c00700 1 -- 172.21.15.27:0/3134715381 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa4940588e0 con 0x7fa4a8105830 2024-04-16T05:54:30.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:54:30.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:54:30.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:30.140+0000 7fa4ae400700 1 -- 172.21.15.27:0/3134715381 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4a81005e0 con 0x7fa49006ec00 2024-04-16T05:54:30.141 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:30.143+0000 7fa4a4c00700 1 -- 172.21.15.27:0/3134715381 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+175 (secure 0 0 0) 0x7fa4a81005e0 con 0x7fa49006ec00 2024-04-16T05:54:30.141 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:54:30.141 INFO:teuthology.orchestra.run.smithi027.stdout: "target_image": null, 2024-04-16T05:54:30.142 INFO:teuthology.orchestra.run.smithi027.stdout: "in_progress": false, 2024-04-16T05:54:30.142 INFO:teuthology.orchestra.run.smithi027.stdout: "which": "", 2024-04-16T05:54:30.142 INFO:teuthology.orchestra.run.smithi027.stdout: "services_complete": [], 2024-04-16T05:54:30.142 INFO:teuthology.orchestra.run.smithi027.stdout: "progress": null, 2024-04-16T05:54:30.142 INFO:teuthology.orchestra.run.smithi027.stdout: "message": "", 2024-04-16T05:54:30.142 INFO:teuthology.orchestra.run.smithi027.stdout: "is_paused": false 2024-04-16T05:54:30.142 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:54:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:30.146+0000 7fa4ae400700 1 -- 172.21.15.27:0/3134715381 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa49006ec00 msgr2=0x7fa4900710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:54:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:30.146+0000 7fa4ae400700 1 --2- 172.21.15.27:0/3134715381 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa49006ec00 0x7fa4900710b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa49c007fd0 tx=0x7fa49c009040 comp rx=0 tx=0).stop 2024-04-16T05:54:30.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:30.146+0000 7fa4ae400700 1 -- 172.21.15.27:0/3134715381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4a8105830 msgr2=0x7fa4a81160c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:54:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:30.146+0000 7fa4ae400700 1 --2- 172.21.15.27:0/3134715381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4a8105830 0x7fa4a81160c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa494009100 tx=0x7fa494009e30 comp rx=0 tx=0).stop 2024-04-16T05:54:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:30.146+0000 7fa4ae400700 1 -- 172.21.15.27:0/3134715381 shutdown_connections 2024-04-16T05:54:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:30.146+0000 7fa4ae400700 1 --2- 172.21.15.27:0/3134715381 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa49006ec00 0x7fa4900710b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:30.146+0000 7fa4ae400700 1 --2- 172.21.15.27:0/3134715381 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa4a8104e90 0x7fa4a8115b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:30.146+0000 7fa4ae400700 1 --2- 172.21.15.27:0/3134715381 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa4a8105830 0x7fa4a81160c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:30.146+0000 7fa4ae400700 1 -- 172.21.15.27:0/3134715381 >> 172.21.15.27:0/3134715381 conn(0x7fa4a80fab10 msgr2=0x7fa4a80fc9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:54:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:30.147+0000 7fa4ae400700 1 -- 172.21.15.27:0/3134715381 shutdown_connections 2024-04-16T05:54:30.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:30.147+0000 7fa4ae400700 1 -- 172.21.15.27:0/3134715381 wait complete. 2024-04-16T05:54:30.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:30 smithi027 ceph-mon[93635]: pgmap v312: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:30.988 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph health detail' 2024-04-16T05:54:31.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:30 smithi122 ceph-mon[80930]: pgmap v312: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:31.525 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:54:31.989 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:31 smithi027 ceph-mon[93635]: from='client.34616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:54:32.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:31 smithi122 ceph-mon[80930]: from='client.34616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:54:32.928 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:32 smithi027 ceph-mon[93635]: pgmap v313: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:33.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:32 smithi122 ceph-mon[80930]: pgmap v313: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:33.822 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.824+0000 7f2d62200700 1 -- 172.21.15.27:0/2123495500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d5c1094c0 msgr2=0x7f2d5c109890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:54:33.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.824+0000 7f2d62200700 1 --2- 172.21.15.27:0/2123495500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d5c1094c0 0x7f2d5c109890 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c004940 tx=0x7f2d4c009e30 comp rx=0 tx=0).stop 2024-04-16T05:54:33.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.824+0000 7f2d62200700 1 -- 172.21.15.27:0/2123495500 shutdown_connections 2024-04-16T05:54:33.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.824+0000 7f2d62200700 1 --2- 172.21.15.27:0/2123495500 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d5c103460 0x7f2d5c1038d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:33.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.824+0000 7f2d62200700 1 --2- 172.21.15.27:0/2123495500 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d5c1094c0 0x7f2d5c109890 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:33.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.824+0000 7f2d62200700 1 -- 172.21.15.27:0/2123495500 >> 172.21.15.27:0/2123495500 conn(0x7f2d5c072c30 msgr2=0x7f2d5c075040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:54:33.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.824+0000 7f2d62200700 1 -- 172.21.15.27:0/2123495500 shutdown_connections 2024-04-16T05:54:33.823 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.824+0000 7f2d62200700 1 -- 172.21.15.27:0/2123495500 wait complete. 2024-04-16T05:54:33.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.825+0000 7f2d62200700 1 Processor -- start 2024-04-16T05:54:33.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.826+0000 7f2d62200700 1 -- start start 2024-04-16T05:54:33.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.826+0000 7f2d62200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d5c103460 0x7f2d5c119f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:54:33.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.826+0000 7f2d62200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d5c1094c0 0x7f2d5c11a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:54:33.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.826+0000 7f2d62200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c11aaa0 con 0x7f2d5c1094c0 2024-04-16T05:54:33.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.826+0000 7f2d62200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5c114080 con 0x7f2d5c103460 2024-04-16T05:54:33.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.827+0000 7f2d5aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d5c1094c0 0x7f2d5c11a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:54:33.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.827+0000 7f2d5aa00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d5c1094c0 0x7f2d5c11a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:35150/0 (socket says 172.21.15.27:35150) 2024-04-16T05:54:33.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.827+0000 7f2d5aa00700 1 -- 172.21.15.27:0/4178987155 learned_addr learned my addr 172.21.15.27:0/4178987155 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:54:33.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.827+0000 7f2d5b400700 1 --2- 172.21.15.27:0/4178987155 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d5c103460 0x7f2d5c119f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:54:33.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.827+0000 7f2d5aa00700 1 -- 172.21.15.27:0/4178987155 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d5c103460 msgr2=0x7f2d5c119f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:54:33.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.827+0000 7f2d5aa00700 1 --2- 172.21.15.27:0/4178987155 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d5c103460 0x7f2d5c119f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:33.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.827+0000 7f2d5aa00700 1 -- 172.21.15.27:0/4178987155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d4c005040 con 0x7f2d5c1094c0 2024-04-16T05:54:33.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.827+0000 7f2d5aa00700 1 --2- 172.21.15.27:0/4178987155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d5c1094c0 0x7f2d5c11a450 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2d4800eb40 tx=0x7f2d4800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:54:33.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.828+0000 7f2d53400700 1 -- 172.21.15.27:0/4178987155 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d48007cc0 con 0x7f2d5c1094c0 2024-04-16T05:54:33.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.828+0000 7f2d53400700 1 -- 172.21.15.27:0/4178987155 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d4800f040 con 0x7f2d5c1094c0 2024-04-16T05:54:33.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.828+0000 7f2d53400700 1 -- 172.21.15.27:0/4178987155 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d48009620 con 0x7f2d5c1094c0 2024-04-16T05:54:33.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.828+0000 7f2d62200700 1 -- 172.21.15.27:0/4178987155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5c114300 con 0x7f2d5c1094c0 2024-04-16T05:54:33.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.828+0000 7f2d62200700 1 -- 172.21.15.27:0/4178987155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d5c114840 con 0x7f2d5c1094c0 2024-04-16T05:54:33.827 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.829+0000 7f2d62200700 1 -- 172.21.15.27:0/4178987155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d5c104ba0 con 0x7f2d5c1094c0 2024-04-16T05:54:33.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.830+0000 7f2d53400700 1 -- 172.21.15.27:0/4178987155 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f2d48009780 con 0x7f2d5c1094c0 2024-04-16T05:54:33.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.832+0000 7f2d53400700 1 --2- 172.21.15.27:0/4178987155 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2d4006ec00 0x7f2d400710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:54:33.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.832+0000 7f2d53400700 1 -- 172.21.15.27:0/4178987155 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f2d4808e6a0 con 0x7f2d5c1094c0 2024-04-16T05:54:33.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.832+0000 7f2d5b400700 1 --2- 172.21.15.27:0/4178987155 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2d4006ec00 0x7f2d400710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:54:33.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.833+0000 7f2d5b400700 1 --2- 172.21.15.27:0/4178987155 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2d4006ec00 0x7f2d400710b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c007f50 tx=0x7f2d4c009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:54:33.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:33.837+0000 7f2d53400700 1 -- 172.21.15.27:0/4178987155 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f2d4805aeb0 con 0x7f2d5c1094c0 2024-04-16T05:54:34.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:34.163+0000 7f2d62200700 1 -- 172.21.15.27:0/4178987155 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "detail": "detail"} v 0) v1 -- 0x7f2d5c04c710 con 0x7f2d5c1094c0 2024-04-16T05:54:34.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:34.164+0000 7f2d53400700 1 -- 172.21.15.27:0/4178987155 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "detail": "detail"}]=0 v0) v1 ==== 74+0+10 (secure 0 0 0) 0x7f2d4805aa40 con 0x7f2d5c1094c0 2024-04-16T05:54:34.162 INFO:teuthology.orchestra.run.smithi027.stdout:HEALTH_OK 2024-04-16T05:54:34.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:34.167+0000 7f2d62200700 1 -- 172.21.15.27:0/4178987155 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2d4006ec00 msgr2=0x7f2d400710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:54:34.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:34.167+0000 7f2d62200700 1 --2- 172.21.15.27:0/4178987155 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2d4006ec00 0x7f2d400710b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c007f50 tx=0x7f2d4c009e30 comp rx=0 tx=0).stop 2024-04-16T05:54:34.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:34.167+0000 7f2d62200700 1 -- 172.21.15.27:0/4178987155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d5c1094c0 msgr2=0x7f2d5c11a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:54:34.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:34.167+0000 7f2d62200700 1 --2- 172.21.15.27:0/4178987155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d5c1094c0 0x7f2d5c11a450 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2d4800eb40 tx=0x7f2d4800b040 comp rx=0 tx=0).stop 2024-04-16T05:54:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:34.168+0000 7f2d62200700 1 -- 172.21.15.27:0/4178987155 shutdown_connections 2024-04-16T05:54:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:34.168+0000 7f2d62200700 1 --2- 172.21.15.27:0/4178987155 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f2d4006ec00 0x7f2d400710b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:34.168+0000 7f2d62200700 1 --2- 172.21.15.27:0/4178987155 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2d5c103460 0x7f2d5c119f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:34.168+0000 7f2d62200700 1 --2- 172.21.15.27:0/4178987155 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d5c1094c0 0x7f2d5c11a450 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:34.168+0000 7f2d62200700 1 -- 172.21.15.27:0/4178987155 >> 172.21.15.27:0/4178987155 conn(0x7f2d5c072c30 msgr2=0x7f2d5c105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:54:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:34.168+0000 7f2d62200700 1 -- 172.21.15.27:0/4178987155 shutdown_connections 2024-04-16T05:54:34.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:34.168+0000 7f2d62200700 1 -- 172.21.15.27:0/4178987155 wait complete. 2024-04-16T05:54:34.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:34 smithi027 ceph-mon[93635]: pgmap v314: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:34.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:34 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/4178987155' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:54:34.991 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph versions' 2024-04-16T05:54:35.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:34 smithi122 ceph-mon[80930]: pgmap v314: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:35.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:34 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/4178987155' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-16T05:54:35.510 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:54:36.008 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:35 smithi027 ceph-mon[93635]: pgmap v315: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:36.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:35 smithi122 ceph-mon[80930]: pgmap v315: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:37.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.791+0000 7fcaeb000700 1 -- 172.21.15.27:0/3284745752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcae4109dc0 msgr2=0x7fcae4103480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:54:37.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.791+0000 7fcaeb000700 1 --2- 172.21.15.27:0/3284745752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcae4109dc0 0x7fcae4103480 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fcad0004940 tx=0x7fcad0009e30 comp rx=0 tx=0).stop 2024-04-16T05:54:37.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.791+0000 7fcaeb000700 1 -- 172.21.15.27:0/3284745752 shutdown_connections 2024-04-16T05:54:37.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.791+0000 7fcaeb000700 1 --2- 172.21.15.27:0/3284745752 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcae4109dc0 0x7fcae4103480 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:37.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.791+0000 7fcaeb000700 1 --2- 172.21.15.27:0/3284745752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcae4109420 0x7fcae41097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:37.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.791+0000 7fcaeb000700 1 -- 172.21.15.27:0/3284745752 >> 172.21.15.27:0/3284745752 conn(0x7fcae4072bd0 msgr2=0x7fcae4074fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:54:37.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.791+0000 7fcaeb000700 1 -- 172.21.15.27:0/3284745752 shutdown_connections 2024-04-16T05:54:37.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.791+0000 7fcaeb000700 1 -- 172.21.15.27:0/3284745752 wait complete. 2024-04-16T05:54:37.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.792+0000 7fcaeb000700 1 Processor -- start 2024-04-16T05:54:37.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.792+0000 7fcaeb000700 1 -- start start 2024-04-16T05:54:37.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.793+0000 7fcaeb000700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcae4109420 0x7fcae4119fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:54:37.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.793+0000 7fcaeb000700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcae4109dc0 0x7fcae411a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:54:37.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.793+0000 7fcaeb000700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae411ab50 con 0x7fcae4109dc0 2024-04-16T05:54:37.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.793+0000 7fcaeb000700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae4114130 con 0x7fcae4109420 2024-04-16T05:54:37.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.793+0000 7fcae3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcae4109dc0 0x7fcae411a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:54:37.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.793+0000 7fcae3400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcae4109dc0 0x7fcae411a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42046/0 (socket says 172.21.15.27:42046) 2024-04-16T05:54:37.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.793+0000 7fcae3400700 1 -- 172.21.15.27:0/4154622540 learned_addr learned my addr 172.21.15.27:0/4154622540 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:54:37.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.794+0000 7fcae3e00700 1 --2- 172.21.15.27:0/4154622540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcae4109420 0x7fcae4119fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:54:37.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.794+0000 7fcae3400700 1 -- 172.21.15.27:0/4154622540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcae4109420 msgr2=0x7fcae4119fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:54:37.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.794+0000 7fcae3400700 1 --2- 172.21.15.27:0/4154622540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcae4109420 0x7fcae4119fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:37.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.794+0000 7fcae3400700 1 -- 172.21.15.27:0/4154622540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad0005040 con 0x7fcae4109dc0 2024-04-16T05:54:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.794+0000 7fcae3400700 1 --2- 172.21.15.27:0/4154622540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcae4109dc0 0x7fcae411a500 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcad0009100 tx=0x7fcad0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:54:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.794+0000 7fcadbe00700 1 -- 172.21.15.27:0/4154622540 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcad000e040 con 0x7fcae4109dc0 2024-04-16T05:54:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.794+0000 7fcadbe00700 1 -- 172.21.15.27:0/4154622540 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcad0004290 con 0x7fcae4109dc0 2024-04-16T05:54:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.794+0000 7fcadbe00700 1 -- 172.21.15.27:0/4154622540 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcad001c3f0 con 0x7fcae4109dc0 2024-04-16T05:54:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.794+0000 7fcaeb000700 1 -- 172.21.15.27:0/4154622540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcae41143b0 con 0x7fcae4109dc0 2024-04-16T05:54:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.795+0000 7fcaeb000700 1 -- 172.21.15.27:0/4154622540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcae4114970 con 0x7fcae4109dc0 2024-04-16T05:54:37.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.796+0000 7fcaeb000700 1 -- 172.21.15.27:0/4154622540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcae4104b70 con 0x7fcae4109dc0 2024-04-16T05:54:37.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.797+0000 7fcadbe00700 1 -- 172.21.15.27:0/4154622540 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fcad0013790 con 0x7fcae4109dc0 2024-04-16T05:54:37.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.798+0000 7fcadbe00700 1 --2- 172.21.15.27:0/4154622540 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fcac806ec00 0x7fcac80710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:54:37.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.798+0000 7fcadbe00700 1 -- 172.21.15.27:0/4154622540 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7fcad008fd50 con 0x7fcae4109dc0 2024-04-16T05:54:37.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.798+0000 7fcae3e00700 1 --2- 172.21.15.27:0/4154622540 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fcac806ec00 0x7fcac80710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:54:37.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.799+0000 7fcae3e00700 1 --2- 172.21.15.27:0/4154622540 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fcac806ec00 0x7fcac80710b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcad4004d60 tx=0x7fcad400b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:54:37.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:37.803+0000 7fcadbe00700 1 -- 172.21.15.27:0/4154622540 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fcad005c490 con 0x7fcae4109dc0 2024-04-16T05:54:38.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:38.123+0000 7fcaeb000700 1 -- 172.21.15.27:0/4154622540 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fcae4104b70 con 0x7fcae4109dc0 2024-04-16T05:54:38.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:38.124+0000 7fcadbe00700 1 -- 172.21.15.27:0/4154622540 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fcad0013420 con 0x7fcae4109dc0 2024-04-16T05:54:38.123 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:54:38.123 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:54:38.123 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:54:38.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:54:38.123 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:54:38.123 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:54:38.123 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:54:38.123 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:54:38.123 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T05:54:38.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:54:38.124 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:54:38.124 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T05:54:38.124 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:54:38.124 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:54:38.124 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 14 2024-04-16T05:54:38.124 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:54:38.124 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:54:38.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:38.127+0000 7fcaeb000700 1 -- 172.21.15.27:0/4154622540 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fcac806ec00 msgr2=0x7fcac80710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:54:38.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:38.127+0000 7fcaeb000700 1 --2- 172.21.15.27:0/4154622540 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fcac806ec00 0x7fcac80710b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcad4004d60 tx=0x7fcad400b040 comp rx=0 tx=0).stop 2024-04-16T05:54:38.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:38.127+0000 7fcaeb000700 1 -- 172.21.15.27:0/4154622540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcae4109dc0 msgr2=0x7fcae411a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:54:38.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:38.127+0000 7fcaeb000700 1 --2- 172.21.15.27:0/4154622540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcae4109dc0 0x7fcae411a500 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcad0009100 tx=0x7fcad0009e30 comp rx=0 tx=0).stop 2024-04-16T05:54:38.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:38.127+0000 7fcaeb000700 1 -- 172.21.15.27:0/4154622540 shutdown_connections 2024-04-16T05:54:38.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:38.127+0000 7fcaeb000700 1 --2- 172.21.15.27:0/4154622540 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fcac806ec00 0x7fcac80710b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:38.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:38.127+0000 7fcaeb000700 1 --2- 172.21.15.27:0/4154622540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcae4109420 0x7fcae4119fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:38.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:38.127+0000 7fcaeb000700 1 --2- 172.21.15.27:0/4154622540 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcae4109dc0 0x7fcae411a500 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:54:38.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:38.127+0000 7fcaeb000700 1 -- 172.21.15.27:0/4154622540 >> 172.21.15.27:0/4154622540 conn(0x7fcae4072bd0 msgr2=0x7fcae4074cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:54:38.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:38.127+0000 7fcaeb000700 1 -- 172.21.15.27:0/4154622540 shutdown_connections 2024-04-16T05:54:38.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:54:38.128+0000 7fcaeb000700 1 -- 172.21.15.27:0/4154622540 wait complete. 2024-04-16T05:54:38.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:38 smithi122 ceph-mon[80930]: pgmap v316: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:38.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:38 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/4154622540' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:38.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:38 smithi027 ceph-mon[93635]: pgmap v316: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:38.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:38 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/4154622540' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:54:38.975 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-16T05:54:39.502 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:54:40.771 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:40 smithi122 ceph-mon[80930]: pgmap v317: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:40.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:40 smithi027 ceph-mon[93635]: pgmap v317: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:41.723 INFO:teuthology.orchestra.run.smithi027.stdout:wait for servicemap items w/ changing names to refresh 2024-04-16T05:54:42.665 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'sleep 60' 2024-04-16T05:54:42.771 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:42 smithi027 ceph-mon[93635]: pgmap v318: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:43.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:42 smithi122 ceph-mon[80930]: pgmap v318: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:43.194 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:54:44.528 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:44 smithi027 ceph-mon[93635]: pgmap v319: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:44.528 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:54:45.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:44 smithi122 ceph-mon[80930]: pgmap v319: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:45.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:44 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:54:46.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:46 smithi027 ceph-mon[93635]: pgmap v320: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:47.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:46 smithi122 ceph-mon[80930]: pgmap v320: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:48.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:48 smithi027 ceph-mon[93635]: pgmap v321: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:49.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:48 smithi122 ceph-mon[80930]: pgmap v321: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:50.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:50 smithi027 ceph-mon[93635]: pgmap v322: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:51.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:50 smithi122 ceph-mon[80930]: pgmap v322: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:52.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:52 smithi027 ceph-mon[93635]: pgmap v323: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:53.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:52 smithi122 ceph-mon[80930]: pgmap v323: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:54.824 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:54 smithi122 ceph-mon[80930]: pgmap v324: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:55.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:54 smithi027 ceph-mon[93635]: pgmap v324: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:57.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:56 smithi122 ceph-mon[80930]: pgmap v325: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:57.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:56 smithi027 ceph-mon[93635]: pgmap v325: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:58.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:58 smithi027 ceph-mon[93635]: pgmap v326: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:54:59.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:58 smithi122 ceph-mon[80930]: pgmap v326: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:00.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:54:59 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:55:00.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:54:59 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:55:01.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:00 smithi122 ceph-mon[80930]: pgmap v327: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:01.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:00 smithi027 ceph-mon[93635]: pgmap v327: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:02.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:01 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:55:02.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:01 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:55:03.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:02 smithi122 ceph-mon[80930]: pgmap v328: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:03.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:02 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:55:03.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:02 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:55:03.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:02 smithi027 ceph-mon[93635]: pgmap v328: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:03.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:02 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:55:03.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:02 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:55:04.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:55:04.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:55:04.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:03 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:55:04.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:55:04.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:55:04.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:03 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:55:05.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:04 smithi122 ceph-mon[80930]: pgmap v329: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:05.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:04 smithi027 ceph-mon[93635]: pgmap v329: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:07.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:06 smithi122 ceph-mon[80930]: pgmap v330: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:07.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:06 smithi027 ceph-mon[93635]: pgmap v330: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:08.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:08 smithi027 ceph-mon[93635]: pgmap v331: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:09.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:08 smithi122 ceph-mon[80930]: pgmap v331: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:11.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:10 smithi122 ceph-mon[80930]: pgmap v332: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:11.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:10 smithi027 ceph-mon[93635]: pgmap v332: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:13.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:12 smithi122 ceph-mon[80930]: pgmap v333: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:13.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:12 smithi027 ceph-mon[93635]: pgmap v333: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:14 smithi122 ceph-mon[80930]: pgmap v334: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:55:15.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:14 smithi027 ceph-mon[93635]: pgmap v334: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:15.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:55:17.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:16 smithi122 ceph-mon[80930]: pgmap v335: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:17.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:16 smithi027 ceph-mon[93635]: pgmap v335: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:18.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:18 smithi027 ceph-mon[93635]: pgmap v336: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:19.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:18 smithi122 ceph-mon[80930]: pgmap v336: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:21.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:20 smithi122 ceph-mon[80930]: pgmap v337: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:21.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:20 smithi027 ceph-mon[93635]: pgmap v337: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:23.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:22 smithi122 ceph-mon[80930]: pgmap v338: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:23.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:22 smithi027 ceph-mon[93635]: pgmap v338: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:24.988 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:24 smithi027 ceph-mon[93635]: pgmap v339: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:25.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:24 smithi122 ceph-mon[80930]: pgmap v339: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:27.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:26 smithi122 ceph-mon[80930]: pgmap v340: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:27.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:26 smithi027 ceph-mon[93635]: pgmap v340: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:28.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:28 smithi027 ceph-mon[93635]: pgmap v341: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:29.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:28 smithi122 ceph-mon[80930]: pgmap v341: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:30.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:55:30.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:55:31.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:30 smithi122 ceph-mon[80930]: pgmap v342: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:31.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:30 smithi027 ceph-mon[93635]: pgmap v342: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:32.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:31 smithi122 ceph-mon[80930]: pgmap v343: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:32.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:31 smithi027 ceph-mon[93635]: pgmap v343: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:34.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:34 smithi027 ceph-mon[93635]: pgmap v344: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:35.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:34 smithi122 ceph-mon[80930]: pgmap v344: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:36.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:36 smithi027 ceph-mon[93635]: pgmap v345: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:37.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:36 smithi122 ceph-mon[80930]: pgmap v345: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:38.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:38 smithi027 ceph-mon[93635]: pgmap v346: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:39.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:38 smithi122 ceph-mon[80930]: pgmap v346: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:40.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:40 smithi027 ceph-mon[93635]: pgmap v347: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:41.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:40 smithi122 ceph-mon[80930]: pgmap v347: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:42.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:42 smithi027 ceph-mon[93635]: pgmap v348: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:43.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:42 smithi122 ceph-mon[80930]: pgmap v348: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:45.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:44 smithi122 ceph-mon[80930]: pgmap v349: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:45.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:44 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:55:45.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:44 smithi027 ceph-mon[93635]: pgmap v349: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:45.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:55:46.151 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph orch ps' 2024-04-16T05:55:46.699 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:55:46.857 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:46 smithi027 ceph-mon[93635]: pgmap v350: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:47.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:46 smithi122 ceph-mon[80930]: pgmap v350: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:48.727 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:48 smithi027 ceph-mon[93635]: pgmap v351: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:48.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.967+0000 7f9e59400700 1 -- 172.21.15.27:0/2921461471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e540fb620 msgr2=0x7f9e540fba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:55:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.967+0000 7f9e59400700 1 --2- 172.21.15.27:0/2921461471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e540fb620 0x7f9e540fba90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9e40004940 tx=0x7f9e40009e30 comp rx=0 tx=0).stop 2024-04-16T05:55:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.967+0000 7f9e59400700 1 -- 172.21.15.27:0/2921461471 shutdown_connections 2024-04-16T05:55:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.967+0000 7f9e59400700 1 --2- 172.21.15.27:0/2921461471 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e540fb620 0x7f9e540fba90 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:55:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.967+0000 7f9e59400700 1 --2- 172.21.15.27:0/2921461471 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e54102ea0 0x7f9e54103290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:55:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.967+0000 7f9e59400700 1 -- 172.21.15.27:0/2921461471 >> 172.21.15.27:0/2921461471 conn(0x7f9e540714c0 msgr2=0x7f9e540718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:55:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.967+0000 7f9e59400700 1 -- 172.21.15.27:0/2921461471 shutdown_connections 2024-04-16T05:55:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.967+0000 7f9e59400700 1 -- 172.21.15.27:0/2921461471 wait complete. 2024-04-16T05:55:48.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.968+0000 7f9e59400700 1 Processor -- start 2024-04-16T05:55:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.968+0000 7f9e59400700 1 -- start start 2024-04-16T05:55:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.969+0000 7f9e59400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e540fb620 0x7f9e54115b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:55:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.969+0000 7f9e59400700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e54102ea0 0x7f9e541160c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:55:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.969+0000 7f9e59400700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e54116710 con 0x7f9e540fb620 2024-04-16T05:55:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.969+0000 7f9e59400700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5410fcf0 con 0x7f9e54102ea0 2024-04-16T05:55:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.969+0000 7f9e52800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e540fb620 0x7f9e54115b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:55:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.969+0000 7f9e52800700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e540fb620 0x7f9e54115b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46422/0 (socket says 172.21.15.27:46422) 2024-04-16T05:55:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.969+0000 7f9e52800700 1 -- 172.21.15.27:0/2261105801 learned_addr learned my addr 172.21.15.27:0/2261105801 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:55:48.968 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.969+0000 7f9e51e00700 1 --2- 172.21.15.27:0/2261105801 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e54102ea0 0x7f9e541160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:55:48.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.969+0000 7f9e52800700 1 -- 172.21.15.27:0/2261105801 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e54102ea0 msgr2=0x7f9e541160c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:55:48.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.969+0000 7f9e52800700 1 --2- 172.21.15.27:0/2261105801 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e54102ea0 0x7f9e541160c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:55:48.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.970+0000 7f9e52800700 1 -- 172.21.15.27:0/2261105801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e40005040 con 0x7f9e540fb620 2024-04-16T05:55:48.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.970+0000 7f9e52800700 1 --2- 172.21.15.27:0/2261105801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e540fb620 0x7f9e54115b80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9e4400c8e0 tx=0x7f9e440044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:55:48.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.970+0000 7f9e4aa00700 1 -- 172.21.15.27:0/2261105801 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e4400a940 con 0x7f9e540fb620 2024-04-16T05:55:48.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.970+0000 7f9e4aa00700 1 -- 172.21.15.27:0/2261105801 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e4400ce80 con 0x7f9e540fb620 2024-04-16T05:55:48.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.970+0000 7f9e59400700 1 -- 172.21.15.27:0/2261105801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e5410ffd0 con 0x7f9e540fb620 2024-04-16T05:55:48.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.970+0000 7f9e4aa00700 1 -- 172.21.15.27:0/2261105801 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e440113a0 con 0x7f9e540fb620 2024-04-16T05:55:48.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.970+0000 7f9e59400700 1 -- 172.21.15.27:0/2261105801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e54110590 con 0x7f9e540fb620 2024-04-16T05:55:48.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.972+0000 7f9e59400700 1 -- 172.21.15.27:0/2261105801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e54107330 con 0x7f9e540fb620 2024-04-16T05:55:48.970 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.973+0000 7f9e4aa00700 1 -- 172.21.15.27:0/2261105801 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9e44012400 con 0x7f9e540fb620 2024-04-16T05:55:48.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.974+0000 7f9e4aa00700 1 --2- 172.21.15.27:0/2261105801 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9e3806ec00 0x7f9e380710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:55:48.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.974+0000 7f9e4aa00700 1 -- 172.21.15.27:0/2261105801 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f9e4408eb00 con 0x7f9e540fb620 2024-04-16T05:55:48.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.974+0000 7f9e51e00700 1 --2- 172.21.15.27:0/2261105801 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9e3806ec00 0x7f9e380710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:55:48.972 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.975+0000 7f9e51e00700 1 --2- 172.21.15.27:0/2261105801 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9e3806ec00 0x7f9e380710b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9e4000b3b0 tx=0x7f9e40009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:55:48.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:48.979+0000 7f9e4aa00700 1 -- 172.21.15.27:0/2261105801 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9e4405b240 con 0x7f9e540fb620 2024-04-16T05:55:49.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:48 smithi122 ceph-mon[80930]: pgmap v351: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:49.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:49.223+0000 7f9e59400700 1 -- 172.21.15.27:0/2261105801 --> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e54109fa0 con 0x7f9e3806ec00 2024-04-16T05:55:49.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:49.242+0000 7f9e4aa00700 1 -- 172.21.15.27:0/2261105801 <== mgr.34100 v2:172.21.15.27:6800/2004337715 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7f9e54109fa0 con 0x7f9e3806ec00 2024-04-16T05:55:49.241 INFO:teuthology.orchestra.run.smithi027.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-16T05:55:49.241 INFO:teuthology.orchestra.run.smithi027.stdout:alertmanager.smithi027 smithi027 *:9093,9094 running (2m) 117s ago 22m 13.7M - 0.25.0 c8568f914cd2 42321467ded6 2024-04-16T05:55:49.241 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi027 smithi027 running (8m) 117s ago 23m 6665k - 19.0.0-2884-g218a6914 a4e4811815e4 4aa1f0006685 2024-04-16T05:55:49.241 INFO:teuthology.orchestra.run.smithi027.stdout:crash.smithi122 smithi122 running (8m) 3m ago 21m 6656k - 19.0.0-2884-g218a6914 a4e4811815e4 1c1912507d56 2024-04-16T05:55:49.241 INFO:teuthology.orchestra.run.smithi027.stdout:grafana.smithi027 smithi027 *:3000 running (2m) 117s ago 22m 53.0M - 9.4.12 2bacad6d85d8 ec2f3705610a 2024-04-16T05:55:49.241 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.ejoytk smithi027 running (4m) 117s ago 15m 12.1M - 19.0.0-2884-g218a6914 a4e4811815e4 b3c2178c537e 2024-04-16T05:55:49.241 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi027.hkhpuq smithi027 running (5m) 117s ago 15m 21.6M - 19.0.0-2884-g218a6914 a4e4811815e4 14b0c83f2891 2024-04-16T05:55:49.241 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.lojjdp smithi122 running (4m) 3m ago 15m 11.2M - 19.0.0-2884-g218a6914 a4e4811815e4 8ba95adfaaef 2024-04-16T05:55:49.242 INFO:teuthology.orchestra.run.smithi027.stdout:mds.cephfs.smithi122.rulewi smithi122 running (3m) 3m ago 15m 11.1M - 19.0.0-2884-g218a6914 a4e4811815e4 c60e9ca0f1c8 2024-04-16T05:55:49.242 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi027.qgzypi smithi027 *:8443,9283,8765 running (11m) 117s ago 24m 521M - 19.0.0-2884-g218a6914 a4e4811815e4 b69a0b08e12b 2024-04-16T05:55:49.242 INFO:teuthology.orchestra.run.smithi027.stdout:mgr.smithi122.sgokkq smithi122 *:8443,9283,8765 running (10m) 3m ago 21m 454M - 19.0.0-2884-g218a6914 a4e4811815e4 32e27bcd7e42 2024-04-16T05:55:49.242 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi027 smithi027 running (10m) 117s ago 24m 55.6M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 299ca337e4ca 2024-04-16T05:55:49.242 INFO:teuthology.orchestra.run.smithi027.stdout:mon.smithi122 smithi122 running (9m) 3m ago 21m 40.2M 2048M 19.0.0-2884-g218a6914 a4e4811815e4 5db5f3037ed0 2024-04-16T05:55:49.242 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi027 smithi027 *:9100 running (3m) 117s ago 23m 7591k - 1.5.0 0da6a335fe13 f3382c6beacb 2024-04-16T05:55:49.242 INFO:teuthology.orchestra.run.smithi027.stdout:node-exporter.smithi122 smithi122 *:9100 running (3m) 3m ago 21m 3078k - 1.5.0 0da6a335fe13 1f6817821f3a 2024-04-16T05:55:49.242 INFO:teuthology.orchestra.run.smithi027.stdout:osd.0 smithi027 running (8m) 117s ago 20m 47.4M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 cef03b67bc5a 2024-04-16T05:55:49.242 INFO:teuthology.orchestra.run.smithi027.stdout:osd.1 smithi027 running (7m) 117s ago 19m 46.3M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3a6ab1fc6e79 2024-04-16T05:55:49.242 INFO:teuthology.orchestra.run.smithi027.stdout:osd.2 smithi027 running (7m) 117s ago 19m 44.9M 1557M 19.0.0-2884-g218a6914 a4e4811815e4 3524731d7277 2024-04-16T05:55:49.242 INFO:teuthology.orchestra.run.smithi027.stdout:osd.3 smithi122 running (6m) 3m ago 18m 44.7M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 897d3579f929 2024-04-16T05:55:49.242 INFO:teuthology.orchestra.run.smithi027.stdout:osd.4 smithi122 running (6m) 3m ago 17m 42.2M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 fa97758ff7c5 2024-04-16T05:55:49.242 INFO:teuthology.orchestra.run.smithi027.stdout:osd.5 smithi122 running (5m) 3m ago 17m 49.6M 2581M 19.0.0-2884-g218a6914 a4e4811815e4 fd12260a820d 2024-04-16T05:55:49.242 INFO:teuthology.orchestra.run.smithi027.stdout:prometheus.smithi027 smithi027 *:9095 running (3m) 117s ago 22m 51.2M - 2.43.0 a07b618ecd1d a6cddf5443d5 2024-04-16T05:55:49.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:49.246+0000 7f9e59400700 1 -- 172.21.15.27:0/2261105801 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9e3806ec00 msgr2=0x7f9e380710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:55:49.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:49.246+0000 7f9e59400700 1 --2- 172.21.15.27:0/2261105801 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9e3806ec00 0x7f9e380710b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9e4000b3b0 tx=0x7f9e40009e30 comp rx=0 tx=0).stop 2024-04-16T05:55:49.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:49.246+0000 7f9e59400700 1 -- 172.21.15.27:0/2261105801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e540fb620 msgr2=0x7f9e54115b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:55:49.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:49.246+0000 7f9e59400700 1 --2- 172.21.15.27:0/2261105801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e540fb620 0x7f9e54115b80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9e4400c8e0 tx=0x7f9e440044c0 comp rx=0 tx=0).stop 2024-04-16T05:55:49.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:49.246+0000 7f9e59400700 1 -- 172.21.15.27:0/2261105801 shutdown_connections 2024-04-16T05:55:49.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:49.246+0000 7f9e59400700 1 --2- 172.21.15.27:0/2261105801 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9e3806ec00 0x7f9e380710b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:55:49.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:49.246+0000 7f9e59400700 1 --2- 172.21.15.27:0/2261105801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9e540fb620 0x7f9e54115b80 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:55:49.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:49.246+0000 7f9e59400700 1 --2- 172.21.15.27:0/2261105801 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9e54102ea0 0x7f9e541160c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:55:49.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:49.246+0000 7f9e59400700 1 -- 172.21.15.27:0/2261105801 >> 172.21.15.27:0/2261105801 conn(0x7f9e540714c0 msgr2=0x7f9e540fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:55:49.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:49.247+0000 7f9e59400700 1 -- 172.21.15.27:0/2261105801 shutdown_connections 2024-04-16T05:55:49.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:49.247+0000 7f9e59400700 1 -- 172.21.15.27:0/2261105801 wait complete. 2024-04-16T05:55:50.131 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph versions' 2024-04-16T05:55:50.667 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:55:50.848 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:50 smithi027 ceph-mon[93635]: from='client.34628 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:55:50.848 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:50 smithi027 ceph-mon[93635]: pgmap v352: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:51.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:50 smithi122 ceph-mon[80930]: from='client.34628 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-16T05:55:51.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:50 smithi122 ceph-mon[80930]: pgmap v352: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:52.857 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:52 smithi027 ceph-mon[93635]: pgmap v353: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:52.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.918+0000 7fd116200700 1 -- 172.21.15.27:0/1292855481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1101037c0 msgr2=0x7fd1100fce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:55:52.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.918+0000 7fd116200700 1 --2- 172.21.15.27:0/1292855481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1101037c0 0x7fd1100fce30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc004940 tx=0x7fd0fc009e30 comp rx=0 tx=0).stop 2024-04-16T05:55:52.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.918+0000 7fd116200700 1 -- 172.21.15.27:0/1292855481 shutdown_connections 2024-04-16T05:55:52.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.918+0000 7fd116200700 1 --2- 172.21.15.27:0/1292855481 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd1101037c0 0x7fd1100fce30 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:55:52.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.918+0000 7fd116200700 1 --2- 172.21.15.27:0/1292855481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd110102e20 0x7fd1101031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:55:52.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.918+0000 7fd116200700 1 -- 172.21.15.27:0/1292855481 >> 172.21.15.27:0/1292855481 conn(0x7fd1100f89b0 msgr2=0x7fd1100fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:55:52.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.918+0000 7fd116200700 1 -- 172.21.15.27:0/1292855481 shutdown_connections 2024-04-16T05:55:52.918 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.918+0000 7fd116200700 1 -- 172.21.15.27:0/1292855481 wait complete. 2024-04-16T05:55:52.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.920+0000 7fd116200700 1 Processor -- start 2024-04-16T05:55:52.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.920+0000 7fd116200700 1 -- start start 2024-04-16T05:55:52.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.920+0000 7fd116200700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd110102e20 0x7fd110115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:55:52.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.920+0000 7fd116200700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1101037c0 0x7fd1101161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:55:52.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.920+0000 7fd116200700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd110116800 con 0x7fd110102e20 2024-04-16T05:55:52.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.920+0000 7fd116200700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11010fde0 con 0x7fd1101037c0 2024-04-16T05:55:52.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.921+0000 7fd10f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd110102e20 0x7fd110115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:55:52.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.921+0000 7fd10f400700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd110102e20 0x7fd110115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46436/0 (socket says 172.21.15.27:46436) 2024-04-16T05:55:52.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.921+0000 7fd10f400700 1 -- 172.21.15.27:0/1490665357 learned_addr learned my addr 172.21.15.27:0/1490665357 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:55:52.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.921+0000 7fd10f400700 1 -- 172.21.15.27:0/1490665357 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1101037c0 msgr2=0x7fd1101161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:55:52.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.921+0000 7fd10f400700 1 --2- 172.21.15.27:0/1490665357 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1101037c0 0x7fd1101161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:55:52.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.921+0000 7fd10f400700 1 -- 172.21.15.27:0/1490665357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0fc005040 con 0x7fd110102e20 2024-04-16T05:55:52.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.921+0000 7fd10f400700 1 --2- 172.21.15.27:0/1490665357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd110102e20 0x7fd110115c70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd10000c8c0 tx=0x7fd1000044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:55:52.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.922+0000 7fd107400700 1 -- 172.21.15.27:0/1490665357 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd10000a980 con 0x7fd110102e20 2024-04-16T05:55:52.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.922+0000 7fd107400700 1 -- 172.21.15.27:0/1490665357 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd10000ce60 con 0x7fd110102e20 2024-04-16T05:55:52.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.922+0000 7fd116200700 1 -- 172.21.15.27:0/1490665357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1101100c0 con 0x7fd110102e20 2024-04-16T05:55:52.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.922+0000 7fd107400700 1 -- 172.21.15.27:0/1490665357 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1000115e0 con 0x7fd110102e20 2024-04-16T05:55:52.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.922+0000 7fd116200700 1 -- 172.21.15.27:0/1490665357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd110110600 con 0x7fd110102e20 2024-04-16T05:55:52.921 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.923+0000 7fd116200700 1 -- 172.21.15.27:0/1490665357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd110102140 con 0x7fd110102e20 2024-04-16T05:55:52.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.924+0000 7fd107400700 1 -- 172.21.15.27:0/1490665357 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd100012400 con 0x7fd110102e20 2024-04-16T05:55:52.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.925+0000 7fd107400700 1 --2- 172.21.15.27:0/1490665357 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd0f406ec00 0x7fd0f40710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:55:52.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.925+0000 7fd107400700 1 -- 172.21.15.27:0/1490665357 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7fd10008ec20 con 0x7fd110102e20 2024-04-16T05:55:52.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.926+0000 7fd10ea00700 1 --2- 172.21.15.27:0/1490665357 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd0f406ec00 0x7fd0f40710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:55:52.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.927+0000 7fd10ea00700 1 --2- 172.21.15.27:0/1490665357 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd0f406ec00 0x7fd0f40710b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc004f90 tx=0x7fd0fc009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:55:52.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:52.931+0000 7fd107400700 1 -- 172.21.15.27:0/1490665357 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd10005b2e0 con 0x7fd110102e20 2024-04-16T05:55:53.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:52 smithi122 ceph-mon[80930]: pgmap v353: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:53.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:53.261+0000 7fd116200700 1 -- 172.21.15.27:0/1490665357 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fd110109f00 con 0x7fd110102e20 2024-04-16T05:55:53.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:53.262+0000 7fd107400700 1 -- 172.21.15.27:0/1490665357 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7fd1000126b0 con 0x7fd110102e20 2024-04-16T05:55:53.261 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-04-16T05:55:53.261 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-04-16T05:55:53.261 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:55:53.261 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:55:53.261 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-04-16T05:55:53.262 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 2 2024-04-16T05:55:53.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:55:53.262 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-04-16T05:55:53.262 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 6 2024-04-16T05:55:53.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:55:53.262 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": { 2024-04-16T05:55:53.262 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 4 2024-04-16T05:55:53.262 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-04-16T05:55:53.262 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-04-16T05:55:53.262 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)": 14 2024-04-16T05:55:53.262 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-04-16T05:55:53.262 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-04-16T05:55:53.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:53.265+0000 7fd116200700 1 -- 172.21.15.27:0/1490665357 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd0f406ec00 msgr2=0x7fd0f40710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:55:53.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:53.265+0000 7fd116200700 1 --2- 172.21.15.27:0/1490665357 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd0f406ec00 0x7fd0f40710b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc004f90 tx=0x7fd0fc009e30 comp rx=0 tx=0).stop 2024-04-16T05:55:53.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:53.265+0000 7fd116200700 1 -- 172.21.15.27:0/1490665357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd110102e20 msgr2=0x7fd110115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:55:53.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:53.265+0000 7fd116200700 1 --2- 172.21.15.27:0/1490665357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd110102e20 0x7fd110115c70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd10000c8c0 tx=0x7fd1000044c0 comp rx=0 tx=0).stop 2024-04-16T05:55:53.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:53.265+0000 7fd116200700 1 -- 172.21.15.27:0/1490665357 shutdown_connections 2024-04-16T05:55:53.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:53.265+0000 7fd116200700 1 --2- 172.21.15.27:0/1490665357 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd0f406ec00 0x7fd0f40710b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:55:53.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:53.265+0000 7fd116200700 1 --2- 172.21.15.27:0/1490665357 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd110102e20 0x7fd110115c70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:55:53.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:53.265+0000 7fd116200700 1 --2- 172.21.15.27:0/1490665357 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1101037c0 0x7fd1101161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:55:53.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:53.265+0000 7fd116200700 1 -- 172.21.15.27:0/1490665357 >> 172.21.15.27:0/1490665357 conn(0x7fd1100f89b0 msgr2=0x7fd110070c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:55:53.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:53.266+0000 7fd116200700 1 -- 172.21.15.27:0/1490665357 shutdown_connections 2024-04-16T05:55:53.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:53.266+0000 7fd116200700 1 -- 172.21.15.27:0/1490665357 wait complete. 2024-04-16T05:55:54.011 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:53 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1490665357' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:55:54.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:53 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1490665357' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:55:54.096 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-16T05:55:54.608 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:55:54.923 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:54 smithi122 ceph-mon[80930]: pgmap v354: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:54.934 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:54 smithi027 ceph-mon[93635]: pgmap v354: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:56.615 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:56 smithi027 ceph-mon[93635]: pgmap v355: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:56.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.930+0000 7f6defe00700 1 -- 172.21.15.27:0/1911981778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6de80724f0 msgr2=0x7f6de80728c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:55:56.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.930+0000 7f6defe00700 1 --2- 172.21.15.27:0/1911981778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6de80724f0 0x7f6de80728c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc004940 tx=0x7f6ddc009e30 comp rx=0 tx=0).stop 2024-04-16T05:55:56.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.931+0000 7f6defe00700 1 -- 172.21.15.27:0/1911981778 shutdown_connections 2024-04-16T05:55:56.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.931+0000 7f6defe00700 1 --2- 172.21.15.27:0/1911981778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6de8072e00 0x7f6de810d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:55:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.931+0000 7f6defe00700 1 --2- 172.21.15.27:0/1911981778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6de80724f0 0x7f6de80728c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:55:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.931+0000 7f6defe00700 1 -- 172.21.15.27:0/1911981778 >> 172.21.15.27:0/1911981778 conn(0x7f6de80fabd0 msgr2=0x7f6de80fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:55:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.931+0000 7f6defe00700 1 -- 172.21.15.27:0/1911981778 shutdown_connections 2024-04-16T05:55:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.931+0000 7f6defe00700 1 -- 172.21.15.27:0/1911981778 wait complete. 2024-04-16T05:55:56.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.932+0000 7f6defe00700 1 Processor -- start 2024-04-16T05:55:56.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.932+0000 7f6defe00700 1 -- start start 2024-04-16T05:55:56.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.932+0000 7f6defe00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6de80724f0 0x7f6de8119ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:55:56.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.932+0000 7f6defe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6de8072e00 0x7f6de811a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:55:56.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.932+0000 7f6defe00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de811aa70 con 0x7f6de8072e00 2024-04-16T05:55:56.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.932+0000 7f6defe00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de8114050 con 0x7f6de80724f0 2024-04-16T05:55:56.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.933+0000 7f6deca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6de8072e00 0x7f6de811a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:55:56.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.933+0000 7f6deca00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6de8072e00 0x7f6de811a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56424/0 (socket says 172.21.15.27:56424) 2024-04-16T05:55:56.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.933+0000 7f6deca00700 1 -- 172.21.15.27:0/230822795 learned_addr learned my addr 172.21.15.27:0/230822795 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:55:56.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.933+0000 7f6deca00700 1 -- 172.21.15.27:0/230822795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6de80724f0 msgr2=0x7f6de8119ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:55:56.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.933+0000 7f6ded400700 1 --2- 172.21.15.27:0/230822795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6de80724f0 0x7f6de8119ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:55:56.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.933+0000 7f6deca00700 1 --2- 172.21.15.27:0/230822795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6de80724f0 0x7f6de8119ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:55:56.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.933+0000 7f6deca00700 1 -- 172.21.15.27:0/230822795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ddc005040 con 0x7f6de8072e00 2024-04-16T05:55:56.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.933+0000 7f6ded400700 1 --2- 172.21.15.27:0/230822795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6de80724f0 0x7f6de8119ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:55:56.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.933+0000 7f6deca00700 1 --2- 172.21.15.27:0/230822795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6de8072e00 0x7f6de811a420 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4002a50 tx=0x7f6dd40046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:55:56.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.934+0000 7f6de5600700 1 -- 172.21.15.27:0/230822795 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dd4007cc0 con 0x7f6de8072e00 2024-04-16T05:55:56.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.934+0000 7f6de5600700 1 -- 172.21.15.27:0/230822795 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6dd400f040 con 0x7f6de8072e00 2024-04-16T05:55:56.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.934+0000 7f6defe00700 1 -- 172.21.15.27:0/230822795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de8114330 con 0x7f6de8072e00 2024-04-16T05:55:56.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.934+0000 7f6de5600700 1 -- 172.21.15.27:0/230822795 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dd4009640 con 0x7f6de8072e00 2024-04-16T05:55:56.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.934+0000 7f6defe00700 1 -- 172.21.15.27:0/230822795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de81148f0 con 0x7f6de8072e00 2024-04-16T05:55:56.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.935+0000 7f6defe00700 1 -- 172.21.15.27:0/230822795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de810a5b0 con 0x7f6de8072e00 2024-04-16T05:55:56.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.936+0000 7f6de5600700 1 -- 172.21.15.27:0/230822795 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6dd40097a0 con 0x7f6de8072e00 2024-04-16T05:55:56.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.938+0000 7f6de5600700 1 --2- 172.21.15.27:0/230822795 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6dd006ec00 0x7f6dd00710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:55:56.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.938+0000 7f6de5600700 1 -- 172.21.15.27:0/230822795 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f6dd408e7b0 con 0x7f6de8072e00 2024-04-16T05:55:56.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.938+0000 7f6ded400700 1 --2- 172.21.15.27:0/230822795 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6dd006ec00 0x7f6dd00710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:55:56.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.939+0000 7f6ded400700 1 --2- 172.21.15.27:0/230822795 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6dd006ec00 0x7f6dd00710b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc00b3b0 tx=0x7f6ddc009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:55:56.941 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:56.942+0000 7f6de5600700 1 -- 172.21.15.27:0/230822795 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6dd405af10 con 0x7f6de8072e00 2024-04-16T05:55:57.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:56 smithi122 ceph-mon[80930]: pgmap v355: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:57.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:57.270+0000 7f6defe00700 1 -- 172.21.15.27:0/230822795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6de8115530 con 0x7f6de8072e00 2024-04-16T05:55:57.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:57.272+0000 7f6de5600700 1 -- 172.21.15.27:0/230822795 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7f6dd405aaa0 con 0x7f6de8072e00 2024-04-16T05:55:57.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:57.274+0000 7f6defe00700 1 -- 172.21.15.27:0/230822795 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6dd006ec00 msgr2=0x7f6dd00710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:55:57.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:57.274+0000 7f6defe00700 1 --2- 172.21.15.27:0/230822795 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6dd006ec00 0x7f6dd00710b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc00b3b0 tx=0x7f6ddc009e30 comp rx=0 tx=0).stop 2024-04-16T05:55:57.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:57.274+0000 7f6defe00700 1 -- 172.21.15.27:0/230822795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6de8072e00 msgr2=0x7f6de811a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:55:57.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:57.274+0000 7f6defe00700 1 --2- 172.21.15.27:0/230822795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6de8072e00 0x7f6de811a420 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4002a50 tx=0x7f6dd40046b0 comp rx=0 tx=0).stop 2024-04-16T05:55:57.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:57.274+0000 7f6defe00700 1 -- 172.21.15.27:0/230822795 shutdown_connections 2024-04-16T05:55:57.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:57.274+0000 7f6defe00700 1 --2- 172.21.15.27:0/230822795 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6dd006ec00 0x7f6dd00710b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:55:57.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:57.274+0000 7f6defe00700 1 --2- 172.21.15.27:0/230822795 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6de80724f0 0x7f6de8119ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:55:57.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:57.274+0000 7f6defe00700 1 --2- 172.21.15.27:0/230822795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6de8072e00 0x7f6de811a420 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:55:57.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:57.274+0000 7f6defe00700 1 -- 172.21.15.27:0/230822795 >> 172.21.15.27:0/230822795 conn(0x7f6de80fabd0 msgr2=0x7f6de8107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:55:57.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:57.275+0000 7f6defe00700 1 -- 172.21.15.27:0/230822795 shutdown_connections 2024-04-16T05:55:57.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:55:57.275+0000 7f6defe00700 1 -- 172.21.15.27:0/230822795 wait complete. 2024-04-16T05:55:57.290 INFO:teuthology.orchestra.run.smithi027.stdout:true 2024-04-16T05:55:58.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:57 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/230822795' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:55:58.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:57 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/230822795' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:55:58.168 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -e sha1=218a691417ef57c5d22a8d5cf3c5f41f7b542838 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-16T05:55:58.699 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:55:58.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:58 smithi027 ceph-mon[93635]: pgmap v356: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:59.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:58 smithi122 ceph-mon[80930]: pgmap v356: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:55:59.880 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:55:59 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:56:00.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:55:59 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:56:00.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:00 smithi027 ceph-mon[93635]: pgmap v357: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:01.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.020+0000 7feee5600700 1 -- 172.21.15.27:0/1928602648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feee01035e0 msgr2=0x7feee00fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:01.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.020+0000 7feee5600700 1 --2- 172.21.15.27:0/1928602648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feee01035e0 0x7feee00fcc50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7feecc004940 tx=0x7feecc009e30 comp rx=0 tx=0).stop 2024-04-16T05:56:01.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.020+0000 7feee5600700 1 -- 172.21.15.27:0/1928602648 shutdown_connections 2024-04-16T05:56:01.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.020+0000 7feee5600700 1 --2- 172.21.15.27:0/1928602648 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feee01035e0 0x7feee00fcc50 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:01.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.020+0000 7feee5600700 1 --2- 172.21.15.27:0/1928602648 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feee0102c40 0x7feee0103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:01.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.020+0000 7feee5600700 1 -- 172.21.15.27:0/1928602648 >> 172.21.15.27:0/1928602648 conn(0x7feee00f8790 msgr2=0x7feee00faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:01.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.021+0000 7feee5600700 1 -- 172.21.15.27:0/1928602648 shutdown_connections 2024-04-16T05:56:01.020 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.021+0000 7feee5600700 1 -- 172.21.15.27:0/1928602648 wait complete. 2024-04-16T05:56:01.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.022+0000 7feee5600700 1 Processor -- start 2024-04-16T05:56:01.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.022+0000 7feee5600700 1 -- start start 2024-04-16T05:56:01.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:00 smithi122 ceph-mon[80930]: pgmap v357: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:01.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.023+0000 7feee5600700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feee0102c40 0x7feee0115c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:01.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.023+0000 7feee5600700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feee01035e0 0x7feee0116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:01.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.023+0000 7feee5600700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee01167a0 con 0x7feee0102c40 2024-04-16T05:56:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.023+0000 7feee5600700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee010fb90 con 0x7feee01035e0 2024-04-16T05:56:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.023+0000 7feedfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feee0102c40 0x7feee0115c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.023+0000 7feedfe00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feee0102c40 0x7feee0115c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56444/0 (socket says 172.21.15.27:56444) 2024-04-16T05:56:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.023+0000 7feedfe00700 1 -- 172.21.15.27:0/1491647136 learned_addr learned my addr 172.21.15.27:0/1491647136 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.023+0000 7feedf400700 1 --2- 172.21.15.27:0/1491647136 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feee01035e0 0x7feee0116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.023+0000 7feedfe00700 1 -- 172.21.15.27:0/1491647136 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feee01035e0 msgr2=0x7feee0116150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.023+0000 7feedfe00700 1 --2- 172.21.15.27:0/1491647136 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feee01035e0 0x7feee0116150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.023+0000 7feedfe00700 1 -- 172.21.15.27:0/1491647136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feecc005040 con 0x7feee0102c40 2024-04-16T05:56:01.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.024+0000 7feedfe00700 1 --2- 172.21.15.27:0/1491647136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feee0102c40 0x7feee0115c10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7feed0002a50 tx=0x7feed00046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:01.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.024+0000 7feed7e00700 1 -- 172.21.15.27:0/1491647136 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feed0007cc0 con 0x7feee0102c40 2024-04-16T05:56:01.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.024+0000 7feed7e00700 1 -- 172.21.15.27:0/1491647136 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feed000f040 con 0x7feee0102c40 2024-04-16T05:56:01.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.024+0000 7feed7e00700 1 -- 172.21.15.27:0/1491647136 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feed000db70 con 0x7feee0102c40 2024-04-16T05:56:01.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.024+0000 7feee5600700 1 -- 172.21.15.27:0/1491647136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee010fe70 con 0x7feee0102c40 2024-04-16T05:56:01.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.024+0000 7feee5600700 1 -- 172.21.15.27:0/1491647136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee0110430 con 0x7feee0102c40 2024-04-16T05:56:01.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.026+0000 7feee5600700 1 -- 172.21.15.27:0/1491647136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee00fe390 con 0x7feee0102c40 2024-04-16T05:56:01.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.027+0000 7feed7e00700 1 -- 172.21.15.27:0/1491647136 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7feed000dd10 con 0x7feee0102c40 2024-04-16T05:56:01.026 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.028+0000 7feed7e00700 1 --2- 172.21.15.27:0/1491647136 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7feec406ec00 0x7feec40710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:01.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.028+0000 7feed7e00700 1 -- 172.21.15.27:0/1491647136 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7feed008e140 con 0x7feee0102c40 2024-04-16T05:56:01.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.028+0000 7feedf400700 1 --2- 172.21.15.27:0/1491647136 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7feec406ec00 0x7feec40710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:01.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.029+0000 7feedf400700 1 --2- 172.21.15.27:0/1491647136 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7feec406ec00 0x7feec40710b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7feee0111440 tx=0x7feecc009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:01.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.033+0000 7feed7e00700 1 -- 172.21.15.27:0/1491647136 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7feed005a800 con 0x7feee0102c40 2024-04-16T05:56:01.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.366+0000 7feee5600700 1 -- 172.21.15.27:0/1491647136 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7feee004bed0 con 0x7feee0102c40 2024-04-16T05:56:01.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.368+0000 7feed7e00700 1 -- 172.21.15.27:0/1491647136 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+623 (secure 0 0 0) 0x7feed005a390 con 0x7feee0102c40 2024-04-16T05:56:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.371+0000 7feee5600700 1 -- 172.21.15.27:0/1491647136 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7feec406ec00 msgr2=0x7feec40710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.371+0000 7feee5600700 1 --2- 172.21.15.27:0/1491647136 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7feec406ec00 0x7feec40710b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7feee0111440 tx=0x7feecc009e30 comp rx=0 tx=0).stop 2024-04-16T05:56:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.371+0000 7feee5600700 1 -- 172.21.15.27:0/1491647136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feee0102c40 msgr2=0x7feee0115c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.371+0000 7feee5600700 1 --2- 172.21.15.27:0/1491647136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feee0102c40 0x7feee0115c10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7feed0002a50 tx=0x7feed00046b0 comp rx=0 tx=0).stop 2024-04-16T05:56:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.371+0000 7feee5600700 1 -- 172.21.15.27:0/1491647136 shutdown_connections 2024-04-16T05:56:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.371+0000 7feee5600700 1 --2- 172.21.15.27:0/1491647136 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7feec406ec00 0x7feec40710b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.371+0000 7feee5600700 1 --2- 172.21.15.27:0/1491647136 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feee0102c40 0x7feee0115c10 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.371+0000 7feee5600700 1 --2- 172.21.15.27:0/1491647136 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feee01035e0 0x7feee0116150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.371+0000 7feee5600700 1 -- 172.21.15.27:0/1491647136 >> 172.21.15.27:0/1491647136 conn(0x7feee00f8790 msgr2=0x7feee00f9680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.371+0000 7feee5600700 1 -- 172.21.15.27:0/1491647136 shutdown_connections 2024-04-16T05:56:01.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:01.371+0000 7feee5600700 1 -- 172.21.15.27:0/1491647136 wait complete. 2024-04-16T05:56:01.388 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.0.0-2884-g218a6914 (218a691417ef57c5d22a8d5cf3c5f41f7b542838) squid (dev)" 2024-04-16T05:56:02.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:01 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1491647136' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:56:02.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:01 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1491647136' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-16T05:56:02.201 DEBUG:teuthology.parallel:result is None 2024-04-16T05:56:02.201 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-16T05:56:02.214 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi027.front.sepia.ceph.com 2024-04-16T05:56:02.215 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:wip-64502-quincy-3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-16T05:56:02.734 INFO:teuthology.orchestra.run.smithi027.stderr:Inferring config /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/mon.smithi027/config 2024-04-16T05:56:02.933 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:02 smithi027 ceph-mon[93635]: pgmap v358: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:03.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:02 smithi122 ceph-mon[80930]: pgmap v358: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:04.786 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:04 smithi027 ceph-mon[93635]: pgmap v359: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:04.802 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:04 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:56:04.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.954+0000 7facbce00700 1 -- 172.21.15.27:0/2840132047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facb81026b0 msgr2=0x7facb80fbd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.954+0000 7facbce00700 1 --2- 172.21.15.27:0/2840132047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facb81026b0 0x7facb80fbd20 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7faca4004940 tx=0x7faca4009e30 comp rx=0 tx=0).stop 2024-04-16T05:56:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.954+0000 7facbce00700 1 -- 172.21.15.27:0/2840132047 shutdown_connections 2024-04-16T05:56:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.954+0000 7facbce00700 1 --2- 172.21.15.27:0/2840132047 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facb81026b0 0x7facb80fbd20 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.954+0000 7facbce00700 1 --2- 172.21.15.27:0/2840132047 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7facb8101d10 0x7facb81020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.954+0000 7facbce00700 1 -- 172.21.15.27:0/2840132047 >> 172.21.15.27:0/2840132047 conn(0x7facb8072bc0 msgr2=0x7facb8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.954+0000 7facbce00700 1 -- 172.21.15.27:0/2840132047 shutdown_connections 2024-04-16T05:56:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.954+0000 7facbce00700 1 -- 172.21.15.27:0/2840132047 wait complete. 2024-04-16T05:56:04.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.955+0000 7facbce00700 1 Processor -- start 2024-04-16T05:56:04.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.955+0000 7facbce00700 1 -- start start 2024-04-16T05:56:04.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.956+0000 7facbce00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facb8101d10 0x7facb810d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:04.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.956+0000 7facbce00700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7facb81026b0 0x7facb810b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:04.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.956+0000 7facbce00700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb810e000 con 0x7facb8101d10 2024-04-16T05:56:04.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.956+0000 7facbce00700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb810e170 con 0x7facb81026b0 2024-04-16T05:56:04.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.956+0000 7facb5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facb8101d10 0x7facb810d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:04.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.956+0000 7facb5e00700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facb8101d10 0x7facb810d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33894/0 (socket says 172.21.15.27:33894) 2024-04-16T05:56:04.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.956+0000 7facb5e00700 1 -- 172.21.15.27:0/1233623258 learned_addr learned my addr 172.21.15.27:0/1233623258 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:04.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.956+0000 7facb5400700 1 --2- 172.21.15.27:0/1233623258 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7facb81026b0 0x7facb810b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.957+0000 7facb5e00700 1 -- 172.21.15.27:0/1233623258 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7facb81026b0 msgr2=0x7facb810b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.957+0000 7facb5e00700 1 --2- 172.21.15.27:0/1233623258 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7facb81026b0 0x7facb810b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.957+0000 7facb5e00700 1 -- 172.21.15.27:0/1233623258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca4005040 con 0x7facb8101d10 2024-04-16T05:56:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.957+0000 7facb5400700 1 --2- 172.21.15.27:0/1233623258 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7facb81026b0 0x7facb810b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T05:56:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.957+0000 7facb5e00700 1 --2- 172.21.15.27:0/1233623258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facb8101d10 0x7facb810d970 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7faca800c8c0 tx=0x7faca80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.957+0000 7facae000700 1 -- 172.21.15.27:0/1233623258 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faca800a980 con 0x7facb8101d10 2024-04-16T05:56:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.957+0000 7facbce00700 1 -- 172.21.15.27:0/1233623258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb810bfa0 con 0x7facb8101d10 2024-04-16T05:56:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.957+0000 7facae000700 1 -- 172.21.15.27:0/1233623258 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faca800ce60 con 0x7facb8101d10 2024-04-16T05:56:04.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.957+0000 7facbce00700 1 -- 172.21.15.27:0/1233623258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb810c4e0 con 0x7facb8101d10 2024-04-16T05:56:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.957+0000 7facae000700 1 -- 172.21.15.27:0/1233623258 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faca801b5e0 con 0x7facb8101d10 2024-04-16T05:56:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.959+0000 7facbce00700 1 -- 172.21.15.27:0/1233623258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb80fd460 con 0x7facb8101d10 2024-04-16T05:56:04.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.960+0000 7facae000700 1 -- 172.21.15.27:0/1233623258 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7faca800aae0 con 0x7facb8101d10 2024-04-16T05:56:04.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.961+0000 7facae000700 1 --2- 172.21.15.27:0/1233623258 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fac9c06ec10 0x7fac9c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.961+0000 7facae000700 1 -- 172.21.15.27:0/1233623258 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7faca808eb80 con 0x7facb8101d10 2024-04-16T05:56:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.961+0000 7facb5400700 1 --2- 172.21.15.27:0/1233623258 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fac9c06ec10 0x7fac9c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:04.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.962+0000 7facb5400700 1 --2- 172.21.15.27:0/1233623258 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fac9c06ec10 0x7fac9c0710c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7facb810d250 tx=0x7faca4009e30 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:04.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:04.966+0000 7facae000700 1 -- 172.21.15.27:0/1233623258 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7faca80577b0 con 0x7facb8101d10 2024-04-16T05:56:05.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:04 smithi122 ceph-mon[80930]: pgmap v359: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:05.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:04 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:56:05.255 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:05.257+0000 7facbce00700 1 -- 172.21.15.27:0/1233623258 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7facb804bed0 con 0x7facb8101d10 2024-04-16T05:56:05.256 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:05.257+0000 7facae000700 1 -- 172.21.15.27:0/1233623258 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 36 v36) v1 ==== 76+0+1936 (secure 0 0 0) 0x7faca805ae70 con 0x7facb8101d10 2024-04-16T05:56:05.256 INFO:teuthology.orchestra.run.smithi027.stdout:e36 2024-04-16T05:56:05.256 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-16T05:56:05.256 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-16T05:56:05.256 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: 1 2024-04-16T05:56:05.257 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:05.257 INFO:teuthology.orchestra.run.smithi027.stdout:Filesystem 'cephfs' (1) 2024-04-16T05:56:05.257 INFO:teuthology.orchestra.run.smithi027.stdout:fs_name cephfs 2024-04-16T05:56:05.257 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 35 2024-04-16T05:56:05.257 INFO:teuthology.orchestra.run.smithi027.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-04-16T05:56:05.257 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-04-16T05:40:11.858650+0000 2024-04-16T05:56:05.257 INFO:teuthology.orchestra.run.smithi027.stdout:modified 2024-04-16T05:51:47.041625+0000 2024-04-16T05:56:05.257 INFO:teuthology.orchestra.run.smithi027.stdout:tableserver 0 2024-04-16T05:56:05.257 INFO:teuthology.orchestra.run.smithi027.stdout:root 0 2024-04-16T05:56:05.257 INFO:teuthology.orchestra.run.smithi027.stdout:session_timeout 60 2024-04-16T05:56:05.257 INFO:teuthology.orchestra.run.smithi027.stdout:session_autoclose 300 2024-04-16T05:56:05.257 INFO:teuthology.orchestra.run.smithi027.stdout:max_file_size 1099511627776 2024-04-16T05:56:05.257 INFO:teuthology.orchestra.run.smithi027.stdout:max_xattr_size 65536 2024-04-16T05:56:05.257 INFO:teuthology.orchestra.run.smithi027.stdout:required_client_features {} 2024-04-16T05:56:05.257 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure 0 2024-04-16T05:56:05.258 INFO:teuthology.orchestra.run.smithi027.stdout:last_failure_osd_epoch 88 2024-04-16T05:56:05.258 INFO:teuthology.orchestra.run.smithi027.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-16T05:56:05.258 INFO:teuthology.orchestra.run.smithi027.stdout:max_mds 1 2024-04-16T05:56:05.258 INFO:teuthology.orchestra.run.smithi027.stdout:in 0 2024-04-16T05:56:05.258 INFO:teuthology.orchestra.run.smithi027.stdout:up {0=34400} 2024-04-16T05:56:05.258 INFO:teuthology.orchestra.run.smithi027.stdout:failed 2024-04-16T05:56:05.258 INFO:teuthology.orchestra.run.smithi027.stdout:damaged 2024-04-16T05:56:05.258 INFO:teuthology.orchestra.run.smithi027.stdout:stopped 2024-04-16T05:56:05.258 INFO:teuthology.orchestra.run.smithi027.stdout:data_pools [3] 2024-04-16T05:56:05.258 INFO:teuthology.orchestra.run.smithi027.stdout:metadata_pool 2 2024-04-16T05:56:05.258 INFO:teuthology.orchestra.run.smithi027.stdout:inline_data disabled 2024-04-16T05:56:05.258 INFO:teuthology.orchestra.run.smithi027.stdout:balancer 2024-04-16T05:56:05.258 INFO:teuthology.orchestra.run.smithi027.stdout:bal_rank_mask -1 2024-04-16T05:56:05.259 INFO:teuthology.orchestra.run.smithi027.stdout:standby_count_wanted 1 2024-04-16T05:56:05.259 INFO:teuthology.orchestra.run.smithi027.stdout:qdb_cluster leader: 34400 members: 34400 2024-04-16T05:56:05.259 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.hkhpuq{0:34400} state up:active seq 23 join_fscid=1 addr [v2:172.21.15.27:6826/2831468256,v1:172.21.15.27:6827/2831468256] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:56:05.259 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:05.259 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:05.259 INFO:teuthology.orchestra.run.smithi027.stdout:Standby daemons: 2024-04-16T05:56:05.259 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:05.259 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi027.ejoytk{-1:34432} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.27:6828/3219609959,v1:172.21.15.27:6829/3219609959] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:56:05.259 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.lojjdp{-1:44337} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6824/97103507,v1:172.21.15.122:6825/97103507] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:56:05.259 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.cephfs.smithi122.rulewi{-1:44359} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.122:6826/1030624816,v1:172.21.15.122:6827/1030624816] compat {c=[1],r=[1],i=[1fff]}] 2024-04-16T05:56:05.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:05.260+0000 7facbce00700 1 -- 172.21.15.27:0/1233623258 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fac9c06ec10 msgr2=0x7fac9c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:05.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:05.260+0000 7facbce00700 1 --2- 172.21.15.27:0/1233623258 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fac9c06ec10 0x7fac9c0710c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7facb810d250 tx=0x7faca4009e30 comp rx=0 tx=0).stop 2024-04-16T05:56:05.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:05.260+0000 7facbce00700 1 -- 172.21.15.27:0/1233623258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facb8101d10 msgr2=0x7facb810d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:05.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:05.260+0000 7facbce00700 1 --2- 172.21.15.27:0/1233623258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facb8101d10 0x7facb810d970 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7faca800c8c0 tx=0x7faca80044c0 comp rx=0 tx=0).stop 2024-04-16T05:56:05.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:05.260+0000 7facbce00700 1 -- 172.21.15.27:0/1233623258 shutdown_connections 2024-04-16T05:56:05.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:05.260+0000 7facbce00700 1 --2- 172.21.15.27:0/1233623258 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fac9c06ec10 0x7fac9c0710c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:05.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:05.260+0000 7facbce00700 1 --2- 172.21.15.27:0/1233623258 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facb8101d10 0x7facb810d970 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:05.260+0000 7facbce00700 1 --2- 172.21.15.27:0/1233623258 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7facb81026b0 0x7facb810b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:05.260+0000 7facbce00700 1 -- 172.21.15.27:0/1233623258 >> 172.21.15.27:0/1233623258 conn(0x7facb8072bc0 msgr2=0x7facb80655d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:05.261+0000 7facbce00700 1 -- 172.21.15.27:0/1233623258 shutdown_connections 2024-04-16T05:56:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:05.261+0000 7facbce00700 1 -- 172.21.15.27:0/1233623258 wait complete. 2024-04-16T05:56:05.261 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 36 2024-04-16T05:56:06.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:05 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:56:06.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:05 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:56:06.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:05 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:56:06.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:05 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1233623258' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:56:06.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:05 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:56:06.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:05 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:56:06.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:05 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:56:06.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:05 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1233623258' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-16T05:56:06.323 INFO:teuthology.run_tasks:Running task fs.post_upgrade_checks... 2024-04-16T05:56:06.335 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T05:56:06.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.574+0000 7ff85ea00640 1 -- 172.21.15.27:0/2883434964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff858070120 msgr2=0x7ff8581083d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:06.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.574+0000 7ff85ea00640 1 --2- 172.21.15.27:0/2883434964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff858070120 0x7ff8581083d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff8440048d0 tx=0x7ff844009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:06.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.574+0000 7ff85ea00640 1 -- 172.21.15.27:0/2883434964 shutdown_connections 2024-04-16T05:56:06.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.574+0000 7ff85ea00640 1 --2- 172.21.15.27:0/2883434964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff858070120 0x7ff8581083d0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:06.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.574+0000 7ff85ea00640 1 --2- 172.21.15.27:0/2883434964 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff85806f810 0x7ff85806fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:06.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.574+0000 7ff85ea00640 1 -- 172.21.15.27:0/2883434964 >> 172.21.15.27:0/2883434964 conn(0x7ff8580f7c70 msgr2=0x7ff8580fa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:06.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.574+0000 7ff85ea00640 1 -- 172.21.15.27:0/2883434964 shutdown_connections 2024-04-16T05:56:06.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.574+0000 7ff85ea00640 1 -- 172.21.15.27:0/2883434964 wait complete. 2024-04-16T05:56:06.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.575+0000 7ff85ea00640 1 Processor -- start 2024-04-16T05:56:06.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.576+0000 7ff85ea00640 1 -- start start 2024-04-16T05:56:06.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.576+0000 7ff85ea00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff85806f810 0x7ff858114d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:06.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.576+0000 7ff85ea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff858070120 0x7ff858115250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:06.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.576+0000 7ff85ea00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff858115850 con 0x7ff858070120 2024-04-16T05:56:06.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.576+0000 7ff85ea00640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85810eea0 con 0x7ff85806f810 2024-04-16T05:56:06.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.576+0000 7ff85cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff858070120 0x7ff858115250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:06.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.576+0000 7ff85cc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff858070120 0x7ff858115250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33914/0 (socket says 172.21.15.27:33914) 2024-04-16T05:56:06.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.576+0000 7ff85cc00640 1 -- 172.21.15.27:0/4154049790 learned_addr learned my addr 172.21.15.27:0/4154049790 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:06.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.577+0000 7ff85d600640 1 --2- 172.21.15.27:0/4154049790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff85806f810 0x7ff858114d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:06.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.577+0000 7ff85cc00640 1 -- 172.21.15.27:0/4154049790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff85806f810 msgr2=0x7ff858114d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:06.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.577+0000 7ff85cc00640 1 --2- 172.21.15.27:0/4154049790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff85806f810 0x7ff858114d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:06.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.577+0000 7ff85cc00640 1 -- 172.21.15.27:0/4154049790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff844005040 con 0x7ff858070120 2024-04-16T05:56:06.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.577+0000 7ff85cc00640 1 --2- 172.21.15.27:0/4154049790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff858070120 0x7ff858115250 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff844004930 tx=0x7ff844009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:06.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.578+0000 7ff855600640 1 -- 172.21.15.27:0/4154049790 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8440094c0 con 0x7ff858070120 2024-04-16T05:56:06.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.578+0000 7ff855600640 1 -- 172.21.15.27:0/4154049790 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff844032e50 con 0x7ff858070120 2024-04-16T05:56:06.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.578+0000 7ff85ea00640 1 -- 172.21.15.27:0/4154049790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff85810f120 con 0x7ff858070120 2024-04-16T05:56:06.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.578+0000 7ff855600640 1 -- 172.21.15.27:0/4154049790 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8440433b0 con 0x7ff858070120 2024-04-16T05:56:06.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.578+0000 7ff85ea00640 1 -- 172.21.15.27:0/4154049790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff85810f5e0 con 0x7ff858070120 2024-04-16T05:56:06.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.579+0000 7ff85ea00640 1 -- 172.21.15.27:0/4154049790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff858070f50 con 0x7ff858070120 2024-04-16T05:56:06.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.580+0000 7ff855600640 1 -- 172.21.15.27:0/4154049790 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff84402f070 con 0x7ff858070120 2024-04-16T05:56:06.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.582+0000 7ff855600640 1 --2- 172.21.15.27:0/4154049790 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff84006eb70 0x7ff840071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:06.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.582+0000 7ff855600640 1 -- 172.21.15.27:0/4154049790 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7ff8440b09e0 con 0x7ff858070120 2024-04-16T05:56:06.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.582+0000 7ff85d600640 1 --2- 172.21.15.27:0/4154049790 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff84006eb70 0x7ff840071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:06.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.583+0000 7ff85d600640 1 --2- 172.21.15.27:0/4154049790 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff84006eb70 0x7ff840071020 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff84c0041d0 tx=0x7ff84c009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:06.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.586+0000 7ff855600640 1 -- 172.21.15.27:0/4154049790 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff8440794f0 con 0x7ff858070120 2024-04-16T05:56:06.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.793+0000 7ff85ea00640 1 -- 172.21.15.27:0/4154049790 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff85804bcc0 con 0x7ff858070120 2024-04-16T05:56:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.796+0000 7ff855600640 1 -- 172.21.15.27:0/4154049790 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+17388 (secure 0 0 0) 0x7ff84407cbb0 con 0x7ff858070120 2024-04-16T05:56:06.796 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:06.796 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":88,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","created":"2024-04-16T05:31:01.142297+0000","modified":"2024-04-16T05:51:44.010556+0000","last_up_change":"2024-04-16T05:50:07.322376+0000","last_in_change":"2024-04-16T05:38:07.718747+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T05:36:52.435427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-16T05:40:10.776546+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-16T05:40:11.582658+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4dd7e294-f625-4492-8bd3-18beab34b6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":56,"up_thru":82,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1955571017},{"type":"v1","addr":"172.21.15.27:6803","nonce":1955571017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1955571017},{"type":"v1","addr":"172.21.15.27:6805","nonce":1955571017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1955571017},{"type":"v1","addr":"172.21.15.27:6809","nonce":1955571017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1955571017},{"type":"v1","addr":"172.21.15.27:6807","nonce":1955571017}]},"public_addr":"172.21.15.27:6803/1955571017","cluster_addr":"172.21.15.27:6805/1955571017","heartbeat_back_addr":"172.21.15.27:6809/1955571017","heartbeat_front_addr":"172.21.15.27:6807/1955571017","state":["exists","up"]},{"osd":1,"uuid":"abec61aa-7a28-473b-b9d2-04e8d7de8a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":58,"up_from":62,"up_thru":82,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2379867635},{"type":"v1","addr":"172.21.15.27:6811","nonce":2379867635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2379867635},{"type":"v1","addr":"172.21.15.27:6813","nonce":2379867635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2379867635},{"type":"v1","addr":"172.21.15.27:6817","nonce":2379867635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2379867635},{"type":"v1","addr":"172.21.15.27:6815","nonce":2379867635}]},"public_addr":"172.21.15.27:6811/2379867635","cluster_addr":"172.21.15.27:6813/2379867635","heartbeat_back_addr":"172.21.15.27:6817/2379867635","heartbeat_front_addr":"172.21.15.27:6815/2379867635","state":["exists","up"]},{"osd":2,"uuid":"e50778e4-bf1f-45eb-aeab-05e8dad8fa89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":63,"up_from":67,"up_thru":82,"down_at":64,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1424147666},{"type":"v1","addr":"172.21.15.27:6819","nonce":1424147666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1424147666},{"type":"v1","addr":"172.21.15.27:6821","nonce":1424147666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1424147666},{"type":"v1","addr":"172.21.15.27:6825","nonce":1424147666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1424147666},{"type":"v1","addr":"172.21.15.27:6823","nonce":1424147666}]},"public_addr":"172.21.15.27:6819/1424147666","cluster_addr":"172.21.15.27:6821/1424147666","heartbeat_back_addr":"172.21.15.27:6825/1424147666","heartbeat_front_addr":"172.21.15.27:6823/1424147666","state":["exists","up"]},{"osd":3,"uuid":"8f55dd70-a9d7-4bc1-a79c-028e66571137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":68,"up_from":72,"up_thru":82,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3174341439},{"type":"v1","addr":"172.21.15.122:6801","nonce":3174341439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3174341439},{"type":"v1","addr":"172.21.15.122:6803","nonce":3174341439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3174341439},{"type":"v1","addr":"172.21.15.122:6807","nonce":3174341439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3174341439},{"type":"v1","addr":"172.21.15.122:6805","nonce":3174341439}]},"public_addr":"172.21.15.122:6801/3174341439","cluster_addr":"172.21.15.122:6803/3174341439","heartbeat_back_addr":"172.21.15.122:6807/3174341439","heartbeat_front_addr":"172.21.15.122:6805/3174341439","state":["exists","up"]},{"osd":4,"uuid":"ba8734db-cb7a-4242-b354-8ff1f524f25a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":73,"up_from":77,"up_thru":82,"down_at":74,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2308306674},{"type":"v1","addr":"172.21.15.122:6809","nonce":2308306674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2308306674},{"type":"v1","addr":"172.21.15.122:6811","nonce":2308306674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2308306674},{"type":"v1","addr":"172.21.15.122:6815","nonce":2308306674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2308306674},{"type":"v1","addr":"172.21.15.122:6813","nonce":2308306674}]},"public_addr":"172.21.15.122:6809/2308306674","cluster_addr":"172.21.15.122:6811/2308306674","heartbeat_back_addr":"172.21.15.122:6815/2308306674","heartbeat_front_addr":"172.21.15.122:6813/2308306674","state":["exists","up"]},{"osd":5,"uuid":"6e2753a7-d194-4103-b586-bfef8cbefa2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":78,"up_from":82,"up_thru":82,"down_at":79,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":3634763653},{"type":"v1","addr":"172.21.15.122:6817","nonce":3634763653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":3634763653},{"type":"v1","addr":"172.21.15.122:6819","nonce":3634763653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":3634763653},{"type":"v1","addr":"172.21.15.122:6823","nonce":3634763653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":3634763653},{"type":"v1","addr":"172.21.15.122:6821","nonce":3634763653}]},"public_addr":"172.21.15.122:6817/3634763653","cluster_addr":"172.21.15.122:6819/3634763653","heartbeat_back_addr":"172.21.15.122:6823/3634763653","heartbeat_front_addr":"172.21.15.122:6821/3634763653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:35:28.179537+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:36:08.890364+0000","dead_epoch":58},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:36:48.694394+0000","dead_epoch":63},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:37:21.969883+0000","dead_epoch":68},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:37:55.840648+0000","dead_epoch":73},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:38:29.596986+0000","dead_epoch":78}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":3,"to":2}]},{"pgid":"2.a","mappings":[{"from":3,"to":0}]},{"pgid":"2.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.6","mappings":[{"from":1,"to":5}]},{"pgid":"3.b","mappings":[{"from":1,"to":2}]},{"pgid":"3.d","mappings":[{"from":3,"to":0}]},{"pgid":"3.15","mappings":[{"from":3,"to":2},{"from":4,"to":5}]},{"pgid":"3.1c","mappings":[{"from":4,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.122:6825/194115669":"2024-04-17T05:51:15.646390+0000","172.21.15.122:6824/194115669":"2024-04-17T05:51:15.646390+0000","172.21.15.27:6829/4115272713":"2024-04-17T05:50:45.882787+0000","172.21.15.122:6827/1078307975":"2024-04-17T05:51:44.010470+0000","172.21.15.27:6828/4115272713":"2024-04-17T05:50:45.882787+0000","172.21.15.27:0/3779871471":"2024-04-17T05:45:28.872665+0000","172.21.15.27:0/3077390652":"2024-04-17T05:43:00.762041+0000","172.21.15.27:6800/2779081505":"2024-04-17T05:32:17.529651+0000","172.21.15.27:6801/2173136120":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/3170976796":"2024-04-17T05:31:34.994394+0000","172.21.15.27:6800/2203377971":"2024-04-17T05:43:00.762041+0000","172.21.15.122:6826/1078307975":"2024-04-17T05:51:44.010470+0000","172.21.15.122:0/1743593984":"2024-04-17T05:44:23.662611+0000","172.21.15.27:6801/2779081505":"2024-04-17T05:32:17.529651+0000","172.21.15.27:0/4258676557":"2024-04-17T05:32:17.529651+0000","172.21.15.122:0/575464263":"2024-04-17T05:44:23.662611+0000","172.21.15.27:0/2398184523":"2024-04-17T05:45:28.872665+0000","172.21.15.27:0/4061328916":"2024-04-17T05:43:00.762041+0000","172.21.15.27:0/99903288":"2024-04-17T05:43:00.762041+0000","172.21.15.27:6826/1980301393":"2024-04-17T05:40:29.686229+0000","172.21.15.27:6800/2700787222":"2024-04-17T05:33:49.465983+0000","172.21.15.27:6800/1090905394":"2024-04-17T05:45:28.872665+0000","172.21.15.122:0/2486581297":"2024-04-17T05:44:23.662611+0000","172.21.15.27:6801/2700787222":"2024-04-17T05:33:49.465983+0000","172.21.15.27:6827/1980301393":"2024-04-17T05:40:29.686229+0000","172.21.15.27:0/3086153578":"2024-04-17T05:32:17.529651+0000","172.21.15.122:6828/871482489":"2024-04-17T05:44:23.662611+0000","172.21.15.27:0/1860000032":"2024-04-17T05:33:49.465983+0000","172.21.15.122:6829/871482489":"2024-04-17T05:44:23.662611+0000","172.21.15.27:0/602364106":"2024-04-17T05:33:49.465983+0000","172.21.15.122:0/915233781":"2024-04-17T05:44:23.662611+0000","172.21.15.27:0/3775595668":"2024-04-17T05:32:17.529651+0000","172.21.15.122:0/2407794149":"2024-04-17T05:44:23.662611+0000","172.21.15.27:0/1372853588":"2024-04-17T05:45:28.872665+0000","172.21.15.27:0/2242461930":"2024-04-17T05:33:49.465983+0000","172.21.15.27:0/2783469284":"2024-04-17T05:45:28.872665+0000","172.21.15.27:6800/2173136120":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/3244097830":"2024-04-17T05:45:28.872665+0000","172.21.15.27:6801/2203377971":"2024-04-17T05:43:00.762041+0000","172.21.15.122:0/3869652783":"2024-04-17T05:44:23.662611+0000","172.21.15.27:0/1691369286":"2024-04-17T05:45:28.872665+0000","172.21.15.27:0/1435069943":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/3713909498":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/2938698524":"2024-04-17T05:43:00.762041+0000","172.21.15.27:6801/1090905394":"2024-04-17T05:45:28.872665+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T05:56:06.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.799+0000 7ff85ea00640 1 -- 172.21.15.27:0/4154049790 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff84006eb70 msgr2=0x7ff840071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:06.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.799+0000 7ff85ea00640 1 --2- 172.21.15.27:0/4154049790 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff84006eb70 0x7ff840071020 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff84c0041d0 tx=0x7ff84c009040 comp rx=0 tx=0).stop 2024-04-16T05:56:06.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.799+0000 7ff85ea00640 1 -- 172.21.15.27:0/4154049790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff858070120 msgr2=0x7ff858115250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:06.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.799+0000 7ff85ea00640 1 --2- 172.21.15.27:0/4154049790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff858070120 0x7ff858115250 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff844004930 tx=0x7ff844009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:06.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.799+0000 7ff85ea00640 1 -- 172.21.15.27:0/4154049790 shutdown_connections 2024-04-16T05:56:06.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.799+0000 7ff85ea00640 1 --2- 172.21.15.27:0/4154049790 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff84006eb70 0x7ff840071020 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:06.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.799+0000 7ff85ea00640 1 --2- 172.21.15.27:0/4154049790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff858070120 0x7ff858115250 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:06.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.799+0000 7ff85ea00640 1 --2- 172.21.15.27:0/4154049790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff85806f810 0x7ff858114d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:06.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.799+0000 7ff85ea00640 1 -- 172.21.15.27:0/4154049790 >> 172.21.15.27:0/4154049790 conn(0x7ff8580f7c70 msgr2=0x7ff858104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.799+0000 7ff85ea00640 1 -- 172.21.15.27:0/4154049790 shutdown_connections 2024-04-16T05:56:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:06.799+0000 7ff85ea00640 1 -- 172.21.15.27:0/4154049790 wait complete. 2024-04-16T05:56:06.819 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T05:36:52.435427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-16T05:40:10.776546+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-16T05:40:11.582658+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T05:56:06.820 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T05:56:07.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:06 smithi122 ceph-mon[80930]: pgmap v360: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:07.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.042+0000 7f7fbec00640 1 -- 172.21.15.27:0/3782324509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fb806f810 msgr2=0x7f7fb806fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:07.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.042+0000 7f7fbec00640 1 --2- 172.21.15.27:0/3782324509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fb806f810 0x7f7fb806fbe0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7fac009c50 tx=0x7f7fac004d10 comp rx=0 tx=0).stop 2024-04-16T05:56:07.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.042+0000 7f7fbec00640 1 -- 172.21.15.27:0/3782324509 shutdown_connections 2024-04-16T05:56:07.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.042+0000 7f7fbec00640 1 --2- 172.21.15.27:0/3782324509 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fb8070120 0x7f7fb81083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:07.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.042+0000 7f7fbec00640 1 --2- 172.21.15.27:0/3782324509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fb806f810 0x7f7fb806fbe0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:07.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.042+0000 7f7fbec00640 1 -- 172.21.15.27:0/3782324509 >> 172.21.15.27:0/3782324509 conn(0x7f7fb80f7c70 msgr2=0x7f7fb80fa080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:07.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.042+0000 7f7fbec00640 1 -- 172.21.15.27:0/3782324509 shutdown_connections 2024-04-16T05:56:07.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.042+0000 7f7fbec00640 1 -- 172.21.15.27:0/3782324509 wait complete. 2024-04-16T05:56:07.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.043+0000 7f7fbec00640 1 Processor -- start 2024-04-16T05:56:07.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.043+0000 7f7fbec00640 1 -- start start 2024-04-16T05:56:07.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.044+0000 7f7fbec00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fb806f810 0x7f7fb8114da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:07.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.044+0000 7f7fbec00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fb8070120 0x7f7fb81152e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:07.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.044+0000 7f7fbec00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb81158e0 con 0x7f7fb8070120 2024-04-16T05:56:07.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.044+0000 7f7fbec00640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb8115a50 con 0x7f7fb806f810 2024-04-16T05:56:07.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.044+0000 7f7fbce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fb8070120 0x7f7fb81152e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.044+0000 7f7fbce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fb8070120 0x7f7fb81152e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33936/0 (socket says 172.21.15.27:33936) 2024-04-16T05:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.044+0000 7f7fbce00640 1 -- 172.21.15.27:0/564922493 learned_addr learned my addr 172.21.15.27:0/564922493 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.044+0000 7f7fbd800640 1 --2- 172.21.15.27:0/564922493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fb806f810 0x7f7fb8114da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.044+0000 7f7fbce00640 1 -- 172.21.15.27:0/564922493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fb806f810 msgr2=0x7f7fb8114da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.044+0000 7f7fbce00640 1 --2- 172.21.15.27:0/564922493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fb806f810 0x7f7fb8114da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.045+0000 7f7fbce00640 1 -- 172.21.15.27:0/564922493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac0098b0 con 0x7f7fb8070120 2024-04-16T05:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.045+0000 7f7fbd800640 1 --2- 172.21.15.27:0/564922493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fb806f810 0x7f7fb8114da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.045+0000 7f7fbce00640 1 --2- 172.21.15.27:0/564922493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fb8070120 0x7f7fb81152e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4002a50 tx=0x7f7fa4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.045+0000 7f7fb5600640 1 -- 172.21.15.27:0/564922493 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fa4007c30 con 0x7f7fb8070120 2024-04-16T05:56:07.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.045+0000 7f7fb5600640 1 -- 172.21.15.27:0/564922493 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7fa400f040 con 0x7f7fb8070120 2024-04-16T05:56:07.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.045+0000 7f7fbec00640 1 -- 172.21.15.27:0/564922493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb810f0e0 con 0x7f7fb8070120 2024-04-16T05:56:07.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.045+0000 7f7fb5600640 1 -- 172.21.15.27:0/564922493 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fa400db50 con 0x7f7fb8070120 2024-04-16T05:56:07.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.046+0000 7f7fbec00640 1 -- 172.21.15.27:0/564922493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb810f600 con 0x7f7fb8070120 2024-04-16T05:56:07.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.047+0000 7f7fbec00640 1 -- 172.21.15.27:0/564922493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb8070f50 con 0x7f7fb8070120 2024-04-16T05:56:07.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.048+0000 7f7fb5600640 1 -- 172.21.15.27:0/564922493 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7fa401c400 con 0x7f7fb8070120 2024-04-16T05:56:07.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.049+0000 7f7fb5600640 1 --2- 172.21.15.27:0/564922493 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7fa006eb70 0x7f7fa0071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:07.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.050+0000 7f7fb5600640 1 -- 172.21.15.27:0/564922493 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f7fa408e390 con 0x7f7fb8070120 2024-04-16T05:56:07.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.050+0000 7f7fbd800640 1 --2- 172.21.15.27:0/564922493 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7fa006eb70 0x7f7fa0071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:07.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.051+0000 7f7fbd800640 1 --2- 172.21.15.27:0/564922493 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7fa006eb70 0x7f7fa0071020 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7fac004a30 tx=0x7f7fac02e040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:07.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.054+0000 7f7fb5600640 1 -- 172.21.15.27:0/564922493 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7fa4056f40 con 0x7f7fb8070120 2024-04-16T05:56:07.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:06 smithi027 ceph-mon[93635]: pgmap v360: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:07.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.264+0000 7f7fbec00640 1 -- 172.21.15.27:0/564922493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7fb810f820 con 0x7f7fb8070120 2024-04-16T05:56:07.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.265+0000 7f7fb5600640 1 -- 172.21.15.27:0/564922493 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7fa405a600 con 0x7f7fb8070120 2024-04-16T05:56:07.264 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-16T05:56:07.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.268+0000 7f7fbec00640 1 -- 172.21.15.27:0/564922493 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7fa006eb70 msgr2=0x7f7fa0071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:07.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.268+0000 7f7fbec00640 1 --2- 172.21.15.27:0/564922493 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7fa006eb70 0x7f7fa0071020 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7fac004a30 tx=0x7f7fac02e040 comp rx=0 tx=0).stop 2024-04-16T05:56:07.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.268+0000 7f7fbec00640 1 -- 172.21.15.27:0/564922493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fb8070120 msgr2=0x7f7fb81152e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:07.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.268+0000 7f7fbec00640 1 --2- 172.21.15.27:0/564922493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fb8070120 0x7f7fb81152e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4002a50 tx=0x7f7fa4004600 comp rx=0 tx=0).stop 2024-04-16T05:56:07.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.268+0000 7f7fbec00640 1 -- 172.21.15.27:0/564922493 shutdown_connections 2024-04-16T05:56:07.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.268+0000 7f7fbec00640 1 --2- 172.21.15.27:0/564922493 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7fa006eb70 0x7f7fa0071020 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:07.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.268+0000 7f7fbec00640 1 --2- 172.21.15.27:0/564922493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fb8070120 0x7f7fb81152e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:07.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.268+0000 7f7fbec00640 1 --2- 172.21.15.27:0/564922493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7fb806f810 0x7f7fb8114da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:07.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.268+0000 7f7fbec00640 1 -- 172.21.15.27:0/564922493 >> 172.21.15.27:0/564922493 conn(0x7f7fb80f7c70 msgr2=0x7f7fb80f9130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:07.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.268+0000 7f7fbec00640 1 -- 172.21.15.27:0/564922493 shutdown_connections 2024-04-16T05:56:07.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.269+0000 7f7fbec00640 1 -- 172.21.15.27:0/564922493 wait complete. 2024-04-16T05:56:07.287 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-16T05:56:07.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.520+0000 7fa877e00640 1 -- 172.21.15.27:0/2132327455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa878102030 msgr2=0x7fa878102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:07.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.520+0000 7fa877e00640 1 --2- 172.21.15.27:0/2132327455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa878102030 0x7fa878102400 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa8680048d0 tx=0x7fa868009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:07.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.520+0000 7fa877e00640 1 -- 172.21.15.27:0/2132327455 shutdown_connections 2024-04-16T05:56:07.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.520+0000 7fa877e00640 1 --2- 172.21.15.27:0/2132327455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8781029d0 0x7fa8780fc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:07.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.520+0000 7fa877e00640 1 --2- 172.21.15.27:0/2132327455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa878102030 0x7fa878102400 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:07.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.520+0000 7fa877e00640 1 -- 172.21.15.27:0/2132327455 >> 172.21.15.27:0/2132327455 conn(0x7fa8780f7c20 msgr2=0x7fa8780fa030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:07.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.520+0000 7fa877e00640 1 -- 172.21.15.27:0/2132327455 shutdown_connections 2024-04-16T05:56:07.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.520+0000 7fa877e00640 1 -- 172.21.15.27:0/2132327455 wait complete. 2024-04-16T05:56:07.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.521+0000 7fa877e00640 1 Processor -- start 2024-04-16T05:56:07.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.521+0000 7fa877e00640 1 -- start start 2024-04-16T05:56:07.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.521+0000 7fa877e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa878102030 0x7fa878114d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:07.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.521+0000 7fa877e00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8781029d0 0x7fa8781152a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:07.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.521+0000 7fa877e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8781158a0 con 0x7fa878102030 2024-04-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.521+0000 7fa877e00640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87810eef0 con 0x7fa8781029d0 2024-04-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.522+0000 7fa876a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa878102030 0x7fa878114d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.522+0000 7fa876a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa878102030 0x7fa878114d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33948/0 (socket says 172.21.15.27:33948) 2024-04-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.522+0000 7fa876a00640 1 -- 172.21.15.27:0/701175717 learned_addr learned my addr 172.21.15.27:0/701175717 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.522+0000 7fa876000640 1 --2- 172.21.15.27:0/701175717 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8781029d0 0x7fa8781152a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.522+0000 7fa876a00640 1 -- 172.21.15.27:0/701175717 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8781029d0 msgr2=0x7fa8781152a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.522+0000 7fa876a00640 1 --2- 172.21.15.27:0/701175717 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8781029d0 0x7fa8781152a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.522+0000 7fa876a00640 1 -- 172.21.15.27:0/701175717 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa868005040 con 0x7fa878102030 2024-04-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.522+0000 7fa876a00640 1 --2- 172.21.15.27:0/701175717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa878102030 0x7fa878114d60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa8680047f0 tx=0x7fa868009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.523+0000 7fa86ea00640 1 -- 172.21.15.27:0/701175717 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8680094c0 con 0x7fa878102030 2024-04-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.523+0000 7fa86ea00640 1 -- 172.21.15.27:0/701175717 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa868032e50 con 0x7fa878102030 2024-04-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.523+0000 7fa877e00640 1 -- 172.21.15.27:0/701175717 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87810f170 con 0x7fa878102030 2024-04-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.523+0000 7fa86ea00640 1 -- 172.21.15.27:0/701175717 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8680433b0 con 0x7fa878102030 2024-04-16T05:56:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.523+0000 7fa877e00640 1 -- 172.21.15.27:0/701175717 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa87810f630 con 0x7fa878102030 2024-04-16T05:56:07.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.524+0000 7fa877e00640 1 -- 172.21.15.27:0/701175717 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8780fd7a0 con 0x7fa878102030 2024-04-16T05:56:07.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.525+0000 7fa86ea00640 1 -- 172.21.15.27:0/701175717 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fa86802f070 con 0x7fa878102030 2024-04-16T05:56:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.526+0000 7fa86ea00640 1 --2- 172.21.15.27:0/701175717 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa85c06eb80 0x7fa85c071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.526+0000 7fa86ea00640 1 -- 172.21.15.27:0/701175717 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7fa8680b0980 con 0x7fa878102030 2024-04-16T05:56:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.527+0000 7fa876000640 1 --2- 172.21.15.27:0/701175717 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa85c06eb80 0x7fa85c071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:07.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.527+0000 7fa876000640 1 --2- 172.21.15.27:0/701175717 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa85c06eb80 0x7fa85c071030 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa864007fd0 tx=0x7fa864009e90 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:07.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.531+0000 7fa86ea00640 1 -- 172.21.15.27:0/701175717 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fa868079490 con 0x7fa878102030 2024-04-16T05:56:07.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.737+0000 7fa877e00640 1 -- 172.21.15.27:0/701175717 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fa8781103d0 con 0x7fa878102030 2024-04-16T05:56:07.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.738+0000 7fa86ea00640 1 -- 172.21.15.27:0/701175717 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v88) v1 ==== 107+0+11 (secure 0 0 0) 0x7fa86807cb50 con 0x7fa878102030 2024-04-16T05:56:07.737 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-16T05:56:07.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.741+0000 7fa877e00640 1 -- 172.21.15.27:0/701175717 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa85c06eb80 msgr2=0x7fa85c071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:07.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.741+0000 7fa877e00640 1 --2- 172.21.15.27:0/701175717 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa85c06eb80 0x7fa85c071030 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa864007fd0 tx=0x7fa864009e90 comp rx=0 tx=0).stop 2024-04-16T05:56:07.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.741+0000 7fa877e00640 1 -- 172.21.15.27:0/701175717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa878102030 msgr2=0x7fa878114d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:07.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.741+0000 7fa877e00640 1 --2- 172.21.15.27:0/701175717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa878102030 0x7fa878114d60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa8680047f0 tx=0x7fa868009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:07.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.741+0000 7fa877e00640 1 -- 172.21.15.27:0/701175717 shutdown_connections 2024-04-16T05:56:07.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.741+0000 7fa877e00640 1 --2- 172.21.15.27:0/701175717 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fa85c06eb80 0x7fa85c071030 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:07.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.741+0000 7fa877e00640 1 --2- 172.21.15.27:0/701175717 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa8781029d0 0x7fa8781152a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:07.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.741+0000 7fa877e00640 1 --2- 172.21.15.27:0/701175717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa878102030 0x7fa878114d60 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.741+0000 7fa877e00640 1 -- 172.21.15.27:0/701175717 >> 172.21.15.27:0/701175717 conn(0x7fa8780f7c20 msgr2=0x7fa878071190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.741+0000 7fa877e00640 1 -- 172.21.15.27:0/701175717 shutdown_connections 2024-04-16T05:56:07.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.741+0000 7fa877e00640 1 -- 172.21.15.27:0/701175717 wait complete. 2024-04-16T05:56:07.758 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-16T05:56:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.990+0000 7fe60ca00640 1 -- 172.21.15.27:0/2561111584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe608104250 msgr2=0x7fe608104620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.990+0000 7fe60ca00640 1 --2- 172.21.15.27:0/2561111584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe608104250 0x7fe608104620 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80048d0 tx=0x7fe5f8009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.991+0000 7fe60ca00640 1 -- 172.21.15.27:0/2561111584 shutdown_connections 2024-04-16T05:56:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.991+0000 7fe60ca00640 1 --2- 172.21.15.27:0/2561111584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe608104bf0 0x7fe6080fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.991+0000 7fe60ca00640 1 --2- 172.21.15.27:0/2561111584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe608104250 0x7fe608104620 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.991+0000 7fe60ca00640 1 -- 172.21.15.27:0/2561111584 >> 172.21.15.27:0/2561111584 conn(0x7fe6080f9fc0 msgr2=0x7fe6080fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.991+0000 7fe60ca00640 1 -- 172.21.15.27:0/2561111584 shutdown_connections 2024-04-16T05:56:07.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.991+0000 7fe60ca00640 1 -- 172.21.15.27:0/2561111584 wait complete. 2024-04-16T05:56:07.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.991+0000 7fe60ca00640 1 Processor -- start 2024-04-16T05:56:07.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.992+0000 7fe60ca00640 1 -- start start 2024-04-16T05:56:07.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.992+0000 7fe60ca00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe608104250 0x7fe608114d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:07.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.992+0000 7fe60ca00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe608104bf0 0x7fe608115270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:07.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.992+0000 7fe60ca00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe608115870 con 0x7fe608104250 2024-04-16T05:56:07.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.992+0000 7fe60ca00640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60810eec0 con 0x7fe608104bf0 2024-04-16T05:56:07.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.992+0000 7fe607400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe608104250 0x7fe608114d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:07.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.992+0000 7fe607400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe608104250 0x7fe608114d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33974/0 (socket says 172.21.15.27:33974) 2024-04-16T05:56:07.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.992+0000 7fe607400640 1 -- 172.21.15.27:0/679710036 learned_addr learned my addr 172.21.15.27:0/679710036 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:07.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.993+0000 7fe607400640 1 -- 172.21.15.27:0/679710036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe608104bf0 msgr2=0x7fe608115270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:07.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.993+0000 7fe606a00640 1 --2- 172.21.15.27:0/679710036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe608104bf0 0x7fe608115270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:07.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.993+0000 7fe607400640 1 --2- 172.21.15.27:0/679710036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe608104bf0 0x7fe608115270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:07.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.993+0000 7fe607400640 1 -- 172.21.15.27:0/679710036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f8005040 con 0x7fe608104250 2024-04-16T05:56:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.993+0000 7fe606a00640 1 --2- 172.21.15.27:0/679710036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe608104bf0 0x7fe608115270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:56:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.993+0000 7fe607400640 1 --2- 172.21.15.27:0/679710036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe608104250 0x7fe608114d30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80047f0 tx=0x7fe5f8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.993+0000 7fe5ff400640 1 -- 172.21.15.27:0/679710036 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5f80094c0 con 0x7fe608104250 2024-04-16T05:56:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.993+0000 7fe5ff400640 1 -- 172.21.15.27:0/679710036 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe5f8032e50 con 0x7fe608104250 2024-04-16T05:56:07.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.993+0000 7fe5ff400640 1 -- 172.21.15.27:0/679710036 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5f80433b0 con 0x7fe608104250 2024-04-16T05:56:07.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.994+0000 7fe60ca00640 1 -- 172.21.15.27:0/679710036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe60810f140 con 0x7fe608104250 2024-04-16T05:56:07.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.994+0000 7fe60ca00640 1 -- 172.21.15.27:0/679710036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60810f600 con 0x7fe608104250 2024-04-16T05:56:07.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.995+0000 7fe60ca00640 1 -- 172.21.15.27:0/679710036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6080ff9c0 con 0x7fe608104250 2024-04-16T05:56:07.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.996+0000 7fe5ff400640 1 -- 172.21.15.27:0/679710036 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe5f802f070 con 0x7fe608104250 2024-04-16T05:56:07.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.997+0000 7fe5ff400640 1 --2- 172.21.15.27:0/679710036 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe5ec06eb80 0x7fe5ec071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:07.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.997+0000 7fe5ff400640 1 -- 172.21.15.27:0/679710036 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7fe5f80b08e0 con 0x7fe608104250 2024-04-16T05:56:07.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.997+0000 7fe606a00640 1 --2- 172.21.15.27:0/679710036 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe5ec06eb80 0x7fe5ec071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:07.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:07.998+0000 7fe606a00640 1 --2- 172.21.15.27:0/679710036 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe5ec06eb80 0x7fe5ec071030 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe6081105f0 tx=0x7fe5f400b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:08.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.002+0000 7fe5ff400640 1 -- 172.21.15.27:0/679710036 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe5f8079490 con 0x7fe608104250 2024-04-16T05:56:08.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:07 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/4154049790' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T05:56:08.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:07 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/564922493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T05:56:08.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:07 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/4154049790' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T05:56:08.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:07 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/564922493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T05:56:08.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.210+0000 7fe60ca00640 1 -- 172.21.15.27:0/679710036 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fe6081103a0 con 0x7fe608104250 2024-04-16T05:56:08.210 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.211+0000 7fe5ff400640 1 -- 172.21.15.27:0/679710036 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v88) v1 ==== 107+0+11 (secure 0 0 0) 0x7fe5f807cb50 con 0x7fe608104250 2024-04-16T05:56:08.210 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-16T05:56:08.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.214+0000 7fe60ca00640 1 -- 172.21.15.27:0/679710036 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe5ec06eb80 msgr2=0x7fe5ec071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:08.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.214+0000 7fe60ca00640 1 --2- 172.21.15.27:0/679710036 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe5ec06eb80 0x7fe5ec071030 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe6081105f0 tx=0x7fe5f400b040 comp rx=0 tx=0).stop 2024-04-16T05:56:08.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.214+0000 7fe60ca00640 1 -- 172.21.15.27:0/679710036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe608104250 msgr2=0x7fe608114d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:08.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.214+0000 7fe60ca00640 1 --2- 172.21.15.27:0/679710036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe608104250 0x7fe608114d30 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80047f0 tx=0x7fe5f8009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:08.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.214+0000 7fe60ca00640 1 -- 172.21.15.27:0/679710036 shutdown_connections 2024-04-16T05:56:08.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.214+0000 7fe60ca00640 1 --2- 172.21.15.27:0/679710036 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe5ec06eb80 0x7fe5ec071030 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:08.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.214+0000 7fe60ca00640 1 --2- 172.21.15.27:0/679710036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe608104bf0 0x7fe608115270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:08.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.214+0000 7fe60ca00640 1 --2- 172.21.15.27:0/679710036 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe608104250 0x7fe608114d30 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:08.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.214+0000 7fe60ca00640 1 -- 172.21.15.27:0/679710036 >> 172.21.15.27:0/679710036 conn(0x7fe6080f9fc0 msgr2=0x7fe6080fbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:08.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.215+0000 7fe60ca00640 1 -- 172.21.15.27:0/679710036 shutdown_connections 2024-04-16T05:56:08.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.215+0000 7fe60ca00640 1 -- 172.21.15.27:0/679710036 wait complete. 2024-04-16T05:56:08.233 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-16T05:56:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.467+0000 7f9c56600640 1 -- 172.21.15.27:0/2317791777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c500fcfe0 msgr2=0x7f9c500fd3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.467+0000 7f9c56600640 1 --2- 172.21.15.27:0/2317791777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c500fcfe0 0x7f9c500fd3b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9c400048d0 tx=0x7f9c40009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.467+0000 7f9c56600640 1 -- 172.21.15.27:0/2317791777 shutdown_connections 2024-04-16T05:56:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.467+0000 7f9c56600640 1 --2- 172.21.15.27:0/2317791777 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c500fd980 0x7f9c50105fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.467+0000 7f9c56600640 1 --2- 172.21.15.27:0/2317791777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c500fcfe0 0x7f9c500fd3b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.467+0000 7f9c56600640 1 -- 172.21.15.27:0/2317791777 >> 172.21.15.27:0/2317791777 conn(0x7f9c500f6ab0 msgr2=0x7f9c500f8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.467+0000 7f9c56600640 1 -- 172.21.15.27:0/2317791777 shutdown_connections 2024-04-16T05:56:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.467+0000 7f9c56600640 1 -- 172.21.15.27:0/2317791777 wait complete. 2024-04-16T05:56:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.468+0000 7f9c56600640 1 Processor -- start 2024-04-16T05:56:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.468+0000 7f9c56600640 1 -- start start 2024-04-16T05:56:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.468+0000 7f9c56600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c500fcfe0 0x7f9c50114d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.468+0000 7f9c56600640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c500fd980 0x7f9c501152b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:08.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.468+0000 7f9c56600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c501158b0 con 0x7f9c500fcfe0 2024-04-16T05:56:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.468+0000 7f9c56600640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c50115a20 con 0x7f9c500fd980 2024-04-16T05:56:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.468+0000 7f9c55200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c500fcfe0 0x7f9c50114d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.468+0000 7f9c55200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c500fcfe0 0x7f9c50114d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33992/0 (socket says 172.21.15.27:33992) 2024-04-16T05:56:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.468+0000 7f9c55200640 1 -- 172.21.15.27:0/1948737626 learned_addr learned my addr 172.21.15.27:0/1948737626 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.469+0000 7f9c47e00640 1 --2- 172.21.15.27:0/1948737626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c500fd980 0x7f9c501152b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.469+0000 7f9c55200640 1 -- 172.21.15.27:0/1948737626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c500fd980 msgr2=0x7f9c501152b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.469+0000 7f9c55200640 1 --2- 172.21.15.27:0/1948737626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c500fd980 0x7f9c501152b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.469+0000 7f9c55200640 1 -- 172.21.15.27:0/1948737626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c40005040 con 0x7f9c500fcfe0 2024-04-16T05:56:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.469+0000 7f9c47e00640 1 --2- 172.21.15.27:0/1948737626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c500fd980 0x7f9c501152b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:56:08.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.469+0000 7f9c55200640 1 --2- 172.21.15.27:0/1948737626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c500fcfe0 0x7f9c50114d70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9c400047f0 tx=0x7f9c40009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:08.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.469+0000 7f9c4d600640 1 -- 172.21.15.27:0/1948737626 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c400094c0 con 0x7f9c500fcfe0 2024-04-16T05:56:08.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.469+0000 7f9c56600640 1 -- 172.21.15.27:0/1948737626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c5010f0c0 con 0x7f9c500fcfe0 2024-04-16T05:56:08.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.469+0000 7f9c4d600640 1 -- 172.21.15.27:0/1948737626 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c40032e50 con 0x7f9c500fcfe0 2024-04-16T05:56:08.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.470+0000 7f9c4d600640 1 -- 172.21.15.27:0/1948737626 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c400433b0 con 0x7f9c500fcfe0 2024-04-16T05:56:08.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.470+0000 7f9c56600640 1 -- 172.21.15.27:0/1948737626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c5010f580 con 0x7f9c500fcfe0 2024-04-16T05:56:08.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.471+0000 7f9c56600640 1 -- 172.21.15.27:0/1948737626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c50065ec0 con 0x7f9c500fcfe0 2024-04-16T05:56:08.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.472+0000 7f9c4d600640 1 -- 172.21.15.27:0/1948737626 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f9c4002f070 con 0x7f9c500fcfe0 2024-04-16T05:56:08.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.473+0000 7f9c4d600640 1 --2- 172.21.15.27:0/1948737626 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9c3406eb80 0x7f9c34071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:08.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.473+0000 7f9c4d600640 1 -- 172.21.15.27:0/1948737626 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f9c400b09e0 con 0x7f9c500fcfe0 2024-04-16T05:56:08.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.473+0000 7f9c47e00640 1 --2- 172.21.15.27:0/1948737626 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9c3406eb80 0x7f9c34071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:08.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.474+0000 7f9c47e00640 1 --2- 172.21.15.27:0/1948737626 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9c3406eb80 0x7f9c34071030 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9c50105c90 tx=0x7f9c3800b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:08.476 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.478+0000 7f9c4d600640 1 -- 172.21.15.27:0/1948737626 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f9c400794f0 con 0x7f9c500fcfe0 2024-04-16T05:56:08.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.727+0000 7f9c56600640 1 -- 172.21.15.27:0/1948737626 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c50109170 con 0x7f9c500fcfe0 2024-04-16T05:56:08.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.729+0000 7f9c4d600640 1 -- 172.21.15.27:0/1948737626 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5207 (secure 0 0 0) 0x7f9c4007cbb0 con 0x7f9c500fcfe0 2024-04-16T05:56:08.728 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:08.728 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34432,"name":"cephfs.smithi027.ejoytk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6829/3219609959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3219609959},{"type":"v1","addr":"172.21.15.27:6829","nonce":3219609959}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":44337,"name":"cephfs.smithi122.lojjdp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6825/97103507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":97103507},{"type":"v1","addr":"172.21.15.122:6825","nonce":97103507}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":44359,"name":"cephfs.smithi122.rulewi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6827/1030624816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1030624816},{"type":"v1","addr":"172.21.15.122:6827","nonce":1030624816}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T05:40:11.858650+0000","modified":"2024-04-16T05:51:47.041625+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34400},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34400":{"gid":34400,"name":"cephfs.smithi027.hkhpuq","rank":0,"incarnation":32,"state":"up:active","state_seq":23,"addr":"172.21.15.27:6827/2831468256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2831468256},{"type":"v1","addr":"172.21.15.27:6827","nonce":2831468256}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":34400,"qdb_cluster":[34400]},"id":1}]} 2024-04-16T05:56:08.728 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 36 2024-04-16T05:56:08.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.731+0000 7f9c56600640 1 -- 172.21.15.27:0/1948737626 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9c3406eb80 msgr2=0x7f9c34071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:08.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.731+0000 7f9c56600640 1 --2- 172.21.15.27:0/1948737626 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9c3406eb80 0x7f9c34071030 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9c50105c90 tx=0x7f9c3800b040 comp rx=0 tx=0).stop 2024-04-16T05:56:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.731+0000 7f9c56600640 1 -- 172.21.15.27:0/1948737626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c500fcfe0 msgr2=0x7f9c50114d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.731+0000 7f9c56600640 1 --2- 172.21.15.27:0/1948737626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c500fcfe0 0x7f9c50114d70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9c400047f0 tx=0x7f9c40009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.731+0000 7f9c56600640 1 -- 172.21.15.27:0/1948737626 shutdown_connections 2024-04-16T05:56:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.732+0000 7f9c56600640 1 --2- 172.21.15.27:0/1948737626 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f9c3406eb80 0x7f9c34071030 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.732+0000 7f9c56600640 1 --2- 172.21.15.27:0/1948737626 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c500fd980 0x7f9c501152b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.732+0000 7f9c56600640 1 --2- 172.21.15.27:0/1948737626 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9c500fcfe0 0x7f9c50114d70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.732+0000 7f9c56600640 1 -- 172.21.15.27:0/1948737626 >> 172.21.15.27:0/1948737626 conn(0x7f9c500f6ab0 msgr2=0x7f9c500f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.732+0000 7f9c56600640 1 -- 172.21.15.27:0/1948737626 shutdown_connections 2024-04-16T05:56:08.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.732+0000 7f9c56600640 1 -- 172.21.15.27:0/1948737626 wait complete. 2024-04-16T05:56:08.751 DEBUG:tasks.fs:checking fs fscid=1,name=cephfs state = {'epoch': 15, 'max_mds': 1, 'flags': 18} 2024-04-16T05:56:08.751 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 16 2024-04-16T05:56:08.982 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.983+0000 7f6ff0400640 1 -- 172.21.15.27:0/2192223822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe8104400 msgr2=0x7f6fe81047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.983+0000 7f6ff0400640 1 --2- 172.21.15.27:0/2192223822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe8104400 0x7f6fe81047d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc0048d0 tx=0x7f6fdc009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.983+0000 7f6ff0400640 1 -- 172.21.15.27:0/2192223822 shutdown_connections 2024-04-16T05:56:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.983+0000 7f6ff0400640 1 --2- 172.21.15.27:0/2192223822 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fe8104da0 0x7f6fe80fe430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.983+0000 7f6ff0400640 1 --2- 172.21.15.27:0/2192223822 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe8104400 0x7f6fe81047d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.983+0000 7f6ff0400640 1 -- 172.21.15.27:0/2192223822 >> 172.21.15.27:0/2192223822 conn(0x7f6fe80f9ff0 msgr2=0x7f6fe80fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.984+0000 7f6ff0400640 1 -- 172.21.15.27:0/2192223822 shutdown_connections 2024-04-16T05:56:08.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.984+0000 7f6ff0400640 1 -- 172.21.15.27:0/2192223822 wait complete. 2024-04-16T05:56:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.985+0000 7f6ff0400640 1 Processor -- start 2024-04-16T05:56:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.985+0000 7f6ff0400640 1 -- start start 2024-04-16T05:56:08.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.985+0000 7f6ff0400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe8104400 0x7f6fe8114ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.985+0000 7f6ff0400640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fe8104da0 0x7f6fe8115420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.985+0000 7f6ff0400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe8115a20 con 0x7f6fe8104400 2024-04-16T05:56:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.985+0000 7f6ff0400640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe810f070 con 0x7f6fe8104da0 2024-04-16T05:56:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.986+0000 7f6feda00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe8104400 0x7f6fe8114ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.986+0000 7f6feda00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe8104400 0x7f6fe8114ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:33998/0 (socket says 172.21.15.27:33998) 2024-04-16T05:56:08.985 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.986+0000 7f6feda00640 1 -- 172.21.15.27:0/1792561580 learned_addr learned my addr 172.21.15.27:0/1792561580 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:08.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.986+0000 7f6fed000640 1 --2- 172.21.15.27:0/1792561580 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fe8104da0 0x7f6fe8115420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:08.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.986+0000 7f6feda00640 1 -- 172.21.15.27:0/1792561580 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fe8104da0 msgr2=0x7f6fe8115420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.986+0000 7f6feda00640 1 --2- 172.21.15.27:0/1792561580 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fe8104da0 0x7f6fe8115420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.987+0000 7f6feda00640 1 -- 172.21.15.27:0/1792561580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fdc005040 con 0x7f6fe8104400 2024-04-16T05:56:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.987+0000 7f6feda00640 1 --2- 172.21.15.27:0/1792561580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe8104400 0x7f6fe8114ee0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc034040 tx=0x7f6fdc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.987+0000 7f6fe5600640 1 -- 172.21.15.27:0/1792561580 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fdc0092b0 con 0x7f6fe8104400 2024-04-16T05:56:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.987+0000 7f6fe5600640 1 -- 172.21.15.27:0/1792561580 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6fdc03b400 con 0x7f6fe8104400 2024-04-16T05:56:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.987+0000 7f6fe5600640 1 -- 172.21.15.27:0/1792561580 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fdc0435d0 con 0x7f6fe8104400 2024-04-16T05:56:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.987+0000 7f6ff0400640 1 -- 172.21.15.27:0/1792561580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe810f2f0 con 0x7f6fe8104400 2024-04-16T05:56:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.988+0000 7f6ff0400640 1 -- 172.21.15.27:0/1792561580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe810f7b0 con 0x7f6fe8104400 2024-04-16T05:56:08.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.989+0000 7f6ff0400640 1 -- 172.21.15.27:0/1792561580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe80ffb70 con 0x7f6fe8104400 2024-04-16T05:56:08.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.990+0000 7f6fe5600640 1 -- 172.21.15.27:0/1792561580 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6fdc02f070 con 0x7f6fe8104400 2024-04-16T05:56:08.989 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.991+0000 7f6fe5600640 1 --2- 172.21.15.27:0/1792561580 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6fd006eb70 0x7f6fd0071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:08.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.991+0000 7f6fe5600640 1 -- 172.21.15.27:0/1792561580 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f6fdc0b07c0 con 0x7f6fe8104400 2024-04-16T05:56:08.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.991+0000 7f6fed000640 1 --2- 172.21.15.27:0/1792561580 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6fd006eb70 0x7f6fd0071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:08.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.992+0000 7f6fed000640 1 --2- 172.21.15.27:0/1792561580 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6fd006eb70 0x7f6fd0071020 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4007fd0 tx=0x7f6fd400b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:08.994 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:08.996+0000 7f6fe5600640 1 -- 172.21.15.27:0/1792561580 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6fdc079350 con 0x7f6fe8104400 2024-04-16T05:56:09.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:08 smithi122 ceph-mon[80930]: pgmap v361: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:09.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:08 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/701175717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T05:56:09.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:08 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/679710036' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T05:56:09.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:08 smithi027 ceph-mon[93635]: pgmap v361: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:09.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:08 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/701175717' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T05:56:09.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:08 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/679710036' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T05:56:09.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.245+0000 7f6ff0400640 1 -- 172.21.15.27:0/1792561580 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 16, "format": "json"} v 0) v1 -- 0x7f6fe810b560 con 0x7f6fe8104400 2024-04-16T05:56:09.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.246+0000 7f6fe5600640 1 -- 172.21.15.27:0/1792561580 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 16, "format": "json"}]=0 dumped fsmap epoch 16 v36) v1 ==== 107+0+4860 (secure 0 0 0) 0x7f6fdc07ca10 con 0x7f6fe8104400 2024-04-16T05:56:09.245 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:09.245 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14474,"name":"cephfs.smithi027.hkhpuq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/2065690722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2065690722},{"type":"v1","addr":"172.21.15.27:6827","nonce":2065690722}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13},{"gid":24263,"name":"cephfs.smithi122.lojjdp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.122:6825/194115669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":194115669},{"type":"v1","addr":"172.21.15.122:6825","nonce":194115669}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":24273,"name":"cephfs.smithi122.rulewi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1078307975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1078307975},{"type":"v1","addr":"172.21.15.122:6827","nonce":1078307975}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T05:40:11.858650+0000","modified":"2024-04-16T05:40:32.701649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14462},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14462":{"gid":14462,"name":"cephfs.smithi027.ejoytk","rank":0,"incarnation":12,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6829/4115272713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4115272713},{"type":"v1","addr":"172.21.15.27:6829","nonce":4115272713}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T05:56:09.246 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2024-04-16T05:56:09.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.249+0000 7f6ff0400640 1 -- 172.21.15.27:0/1792561580 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6fd006eb70 msgr2=0x7f6fd0071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:09.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.249+0000 7f6ff0400640 1 --2- 172.21.15.27:0/1792561580 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6fd006eb70 0x7f6fd0071020 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4007fd0 tx=0x7f6fd400b040 comp rx=0 tx=0).stop 2024-04-16T05:56:09.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.249+0000 7f6ff0400640 1 -- 172.21.15.27:0/1792561580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe8104400 msgr2=0x7f6fe8114ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:09.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.249+0000 7f6ff0400640 1 --2- 172.21.15.27:0/1792561580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe8104400 0x7f6fe8114ee0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc034040 tx=0x7f6fdc009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:09.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.250+0000 7f6ff0400640 1 -- 172.21.15.27:0/1792561580 shutdown_connections 2024-04-16T05:56:09.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.250+0000 7f6ff0400640 1 --2- 172.21.15.27:0/1792561580 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6fd006eb70 0x7f6fd0071020 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:09.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.250+0000 7f6ff0400640 1 --2- 172.21.15.27:0/1792561580 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6fe8104da0 0x7f6fe8115420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:09.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.250+0000 7f6ff0400640 1 --2- 172.21.15.27:0/1792561580 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fe8104400 0x7f6fe8114ee0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:09.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.250+0000 7f6ff0400640 1 -- 172.21.15.27:0/1792561580 >> 172.21.15.27:0/1792561580 conn(0x7f6fe80f9ff0 msgr2=0x7f6fe80faf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:09.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.250+0000 7f6ff0400640 1 -- 172.21.15.27:0/1792561580 shutdown_connections 2024-04-16T05:56:09.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.250+0000 7f6ff0400640 1 -- 172.21.15.27:0/1792561580 wait complete. 2024-04-16T05:56:09.269 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 17 2024-04-16T05:56:09.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.493+0000 7f3fcb200640 1 -- 172.21.15.27:0/2615811173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fcc0faf10 msgr2=0x7f3fcc0fb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:09.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.493+0000 7f3fcb200640 1 --2- 172.21.15.27:0/2615811173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fcc0faf10 0x7f3fcc0fb360 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3fb80048d0 tx=0x7f3fb8009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:09.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.494+0000 7f3fcb200640 1 -- 172.21.15.27:0/2615811173 shutdown_connections 2024-04-16T05:56:09.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.494+0000 7f3fcb200640 1 --2- 172.21.15.27:0/2615811173 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fcc0faf10 0x7f3fcc0fb360 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:09.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.494+0000 7f3fcb200640 1 --2- 172.21.15.27:0/2615811173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fcc100f00 0x7f3fcc1012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:09.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.494+0000 7f3fcb200640 1 -- 172.21.15.27:0/2615811173 >> 172.21.15.27:0/2615811173 conn(0x7f3fcc070f10 msgr2=0x7f3fcc071310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:09.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.494+0000 7f3fcb200640 1 -- 172.21.15.27:0/2615811173 shutdown_connections 2024-04-16T05:56:09.493 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.494+0000 7f3fcb200640 1 -- 172.21.15.27:0/2615811173 wait complete. 2024-04-16T05:56:09.494 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.495+0000 7f3fcb200640 1 Processor -- start 2024-04-16T05:56:09.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.495+0000 7f3fcb200640 1 -- start start 2024-04-16T05:56:09.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.495+0000 7f3fcb200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fcc0faf10 0x7f3fcc101d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:09.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.495+0000 7f3fcb200640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fcc100f00 0x7f3fcc102250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:09.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.495+0000 7f3fcb200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc102870 con 0x7f3fcc0faf10 2024-04-16T05:56:09.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.495+0000 7f3fcb200640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc1029e0 con 0x7f3fcc100f00 2024-04-16T05:56:09.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.495+0000 7f3fc9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fcc0faf10 0x7f3fcc101d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:09.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.495+0000 7f3fc9e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fcc0faf10 0x7f3fcc101d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34028/0 (socket says 172.21.15.27:34028) 2024-04-16T05:56:09.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.496+0000 7f3fc9e00640 1 -- 172.21.15.27:0/1856229583 learned_addr learned my addr 172.21.15.27:0/1856229583 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:09.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.496+0000 7f3fc9400640 1 --2- 172.21.15.27:0/1856229583 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fcc100f00 0x7f3fcc102250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:09.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.496+0000 7f3fc9e00640 1 -- 172.21.15.27:0/1856229583 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fcc100f00 msgr2=0x7f3fcc102250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:09.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.496+0000 7f3fc9e00640 1 --2- 172.21.15.27:0/1856229583 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fcc100f00 0x7f3fcc102250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:09.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.496+0000 7f3fc9e00640 1 -- 172.21.15.27:0/1856229583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb8005040 con 0x7f3fcc0faf10 2024-04-16T05:56:09.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.496+0000 7f3fc9400640 1 --2- 172.21.15.27:0/1856229583 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fcc100f00 0x7f3fcc102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:56:09.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.496+0000 7f3fc9e00640 1 --2- 172.21.15.27:0/1856229583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fcc0faf10 0x7f3fcc101d10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc00b790 tx=0x7f3fbc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:09.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.496+0000 7f3fc2000640 1 -- 172.21.15.27:0/1856229583 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fbc00c700 con 0x7f3fcc0faf10 2024-04-16T05:56:09.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.496+0000 7f3fc2000640 1 -- 172.21.15.27:0/1856229583 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3fbc00cd30 con 0x7f3fcc0faf10 2024-04-16T05:56:09.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.496+0000 7f3fcb200640 1 -- 172.21.15.27:0/1856229583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fcc119040 con 0x7f3fcc0faf10 2024-04-16T05:56:09.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.496+0000 7f3fc2000640 1 -- 172.21.15.27:0/1856229583 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fbc013370 con 0x7f3fcc0faf10 2024-04-16T05:56:09.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.497+0000 7f3fcb200640 1 -- 172.21.15.27:0/1856229583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fcc119500 con 0x7f3fcc0faf10 2024-04-16T05:56:09.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.498+0000 7f3fcb200640 1 -- 172.21.15.27:0/1856229583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fcc0fc600 con 0x7f3fcc0faf10 2024-04-16T05:56:09.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.499+0000 7f3fc2000640 1 -- 172.21.15.27:0/1856229583 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f3fbc012070 con 0x7f3fcc0faf10 2024-04-16T05:56:09.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.500+0000 7f3fc2000640 1 --2- 172.21.15.27:0/1856229583 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3fb006eb80 0x7f3fb0071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:09.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.500+0000 7f3fc2000640 1 -- 172.21.15.27:0/1856229583 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f3fbc08e600 con 0x7f3fcc0faf10 2024-04-16T05:56:09.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.500+0000 7f3fc9400640 1 --2- 172.21.15.27:0/1856229583 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3fb006eb80 0x7f3fb0071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:09.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.501+0000 7f3fc9400640 1 --2- 172.21.15.27:0/1856229583 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3fb006eb80 0x7f3fb0071030 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8003080 tx=0x7f3fb8009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:09.504 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.506+0000 7f3fc2000640 1 -- 172.21.15.27:0/1856229583 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f3fbc057230 con 0x7f3fcc0faf10 2024-04-16T05:56:09.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.754+0000 7f3fcb200640 1 -- 172.21.15.27:0/1856229583 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 17, "format": "json"} v 0) v1 -- 0x7f3fcc103400 con 0x7f3fcc0faf10 2024-04-16T05:56:09.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.755+0000 7f3fc2000640 1 -- 172.21.15.27:0/1856229583 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 17, "format": "json"}]=0 dumped fsmap epoch 17 v36) v1 ==== 107+0+4081 (secure 0 0 0) 0x7f3fbc05a8f0 con 0x7f3fcc0faf10 2024-04-16T05:56:09.754 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:09.754 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24263,"name":"cephfs.smithi122.lojjdp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.122:6825/194115669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":194115669},{"type":"v1","addr":"172.21.15.122:6825","nonce":194115669}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":24273,"name":"cephfs.smithi122.rulewi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1078307975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1078307975},{"type":"v1","addr":"172.21.15.122:6827","nonce":1078307975}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T05:40:11.858650+0000","modified":"2024-04-16T05:40:32.701649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14462},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14462":{"gid":14462,"name":"cephfs.smithi027.ejoytk","rank":0,"incarnation":12,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6829/4115272713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4115272713},{"type":"v1","addr":"172.21.15.27:6829","nonce":4115272713}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T05:56:09.755 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 17 2024-04-16T05:56:09.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.758+0000 7f3fcb200640 1 -- 172.21.15.27:0/1856229583 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3fb006eb80 msgr2=0x7f3fb0071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:09.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.758+0000 7f3fcb200640 1 --2- 172.21.15.27:0/1856229583 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3fb006eb80 0x7f3fb0071030 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8003080 tx=0x7f3fb8009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:09.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.758+0000 7f3fcb200640 1 -- 172.21.15.27:0/1856229583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fcc0faf10 msgr2=0x7f3fcc101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:09.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.758+0000 7f3fcb200640 1 --2- 172.21.15.27:0/1856229583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fcc0faf10 0x7f3fcc101d10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc00b790 tx=0x7f3fbc0091d0 comp rx=0 tx=0).stop 2024-04-16T05:56:09.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.759+0000 7f3fcb200640 1 -- 172.21.15.27:0/1856229583 shutdown_connections 2024-04-16T05:56:09.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.759+0000 7f3fcb200640 1 --2- 172.21.15.27:0/1856229583 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f3fb006eb80 0x7f3fb0071030 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:09.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.759+0000 7f3fcb200640 1 --2- 172.21.15.27:0/1856229583 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3fcc100f00 0x7f3fcc102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:09.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.759+0000 7f3fcb200640 1 --2- 172.21.15.27:0/1856229583 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fcc0faf10 0x7f3fcc101d10 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:09.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.759+0000 7f3fcb200640 1 -- 172.21.15.27:0/1856229583 >> 172.21.15.27:0/1856229583 conn(0x7f3fcc070f10 msgr2=0x7f3fcc064380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:09.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.759+0000 7f3fcb200640 1 -- 172.21.15.27:0/1856229583 shutdown_connections 2024-04-16T05:56:09.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.759+0000 7f3fcb200640 1 -- 172.21.15.27:0/1856229583 wait complete. 2024-04-16T05:56:09.779 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 18 2024-04-16T05:56:09.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.997+0000 7fb322400640 1 -- 172.21.15.27:0/4153104884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31c0fd000 msgr2=0x7fb31c0fd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:09.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.997+0000 7fb322400640 1 --2- 172.21.15.27:0/4153104884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31c0fd000 0x7fb31c0fd3d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb30c0048d0 tx=0x7fb30c009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:09.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.997+0000 7fb322400640 1 -- 172.21.15.27:0/4153104884 shutdown_connections 2024-04-16T05:56:09.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.997+0000 7fb322400640 1 --2- 172.21.15.27:0/4153104884 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb31c0fd9a0 0x7fb31c105fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:09.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.997+0000 7fb322400640 1 --2- 172.21.15.27:0/4153104884 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31c0fd000 0x7fb31c0fd3d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:09.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.997+0000 7fb322400640 1 -- 172.21.15.27:0/4153104884 >> 172.21.15.27:0/4153104884 conn(0x7fb31c0f6a90 msgr2=0x7fb31c0f8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:09.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.997+0000 7fb322400640 1 -- 172.21.15.27:0/4153104884 shutdown_connections 2024-04-16T05:56:09.996 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.997+0000 7fb322400640 1 -- 172.21.15.27:0/4153104884 wait complete. 2024-04-16T05:56:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.998+0000 7fb322400640 1 Processor -- start 2024-04-16T05:56:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.998+0000 7fb322400640 1 -- start start 2024-04-16T05:56:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.998+0000 7fb322400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31c0fd000 0x7fb31c114d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.998+0000 7fb322400640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb31c0fd9a0 0x7fb31c115290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.998+0000 7fb322400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c115890 con 0x7fb31c0fd000 2024-04-16T05:56:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.998+0000 7fb322400640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c115a00 con 0x7fb31c0fd9a0 2024-04-16T05:56:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.998+0000 7fb321000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31c0fd000 0x7fb31c114d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.999+0000 7fb321000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31c0fd000 0x7fb31c114d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34042/0 (socket says 172.21.15.27:34042) 2024-04-16T05:56:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.999+0000 7fb321000640 1 -- 172.21.15.27:0/1493250280 learned_addr learned my addr 172.21.15.27:0/1493250280 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.999+0000 7fb313e00640 1 --2- 172.21.15.27:0/1493250280 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb31c0fd9a0 0x7fb31c115290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.999+0000 7fb321000640 1 -- 172.21.15.27:0/1493250280 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb31c0fd9a0 msgr2=0x7fb31c115290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.999+0000 7fb321000640 1 --2- 172.21.15.27:0/1493250280 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb31c0fd9a0 0x7fb31c115290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:09.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:09.999+0000 7fb321000640 1 -- 172.21.15.27:0/1493250280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb30c005040 con 0x7fb31c0fd000 2024-04-16T05:56:09.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.000+0000 7fb321000640 1 --2- 172.21.15.27:0/1493250280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31c0fd000 0x7fb31c114d50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb30c004930 tx=0x7fb30c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:09.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.000+0000 7fb319600640 1 -- 172.21.15.27:0/1493250280 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb30c0094c0 con 0x7fb31c0fd000 2024-04-16T05:56:09.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.000+0000 7fb319600640 1 -- 172.21.15.27:0/1493250280 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb30c032e50 con 0x7fb31c0fd000 2024-04-16T05:56:09.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.000+0000 7fb319600640 1 -- 172.21.15.27:0/1493250280 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb30c0433b0 con 0x7fb31c0fd000 2024-04-16T05:56:09.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.000+0000 7fb322400640 1 -- 172.21.15.27:0/1493250280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31c10f030 con 0x7fb31c0fd000 2024-04-16T05:56:09.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.000+0000 7fb322400640 1 -- 172.21.15.27:0/1493250280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb31c10f4f0 con 0x7fb31c0fd000 2024-04-16T05:56:10.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.001+0000 7fb322400640 1 -- 172.21.15.27:0/1493250280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31c065e60 con 0x7fb31c0fd000 2024-04-16T05:56:10.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.003+0000 7fb319600640 1 -- 172.21.15.27:0/1493250280 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fb30c02f070 con 0x7fb31c0fd000 2024-04-16T05:56:10.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.004+0000 7fb319600640 1 --2- 172.21.15.27:0/1493250280 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb30006eb70 0x7fb300071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:10.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.004+0000 7fb319600640 1 -- 172.21.15.27:0/1493250280 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7fb30c0b07f0 con 0x7fb31c0fd000 2024-04-16T05:56:10.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.004+0000 7fb313e00640 1 --2- 172.21.15.27:0/1493250280 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb30006eb70 0x7fb300071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:10.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.005+0000 7fb313e00640 1 --2- 172.21.15.27:0/1493250280 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb30006eb70 0x7fb300071020 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb304007fd0 tx=0x7fb304009e90 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:10.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.009+0000 7fb319600640 1 -- 172.21.15.27:0/1493250280 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fb30c0793a0 con 0x7fb31c0fd000 2024-04-16T05:56:10.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:09 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1948737626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T05:56:10.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:09 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1792561580' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 16, "format": "json"}]: dispatch 2024-04-16T05:56:10.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:09 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1948737626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-16T05:56:10.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:09 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1792561580' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 16, "format": "json"}]: dispatch 2024-04-16T05:56:10.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.248+0000 7fb322400640 1 -- 172.21.15.27:0/1493250280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 18, "format": "json"} v 0) v1 -- 0x7fb31c04bcc0 con 0x7fb31c0fd000 2024-04-16T05:56:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.250+0000 7fb319600640 1 -- 172.21.15.27:0/1493250280 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 18, "format": "json"}]=0 dumped fsmap epoch 18 v36) v1 ==== 107+0+4928 (secure 0 0 0) 0x7fb30c07ca60 con 0x7fb31c0fd000 2024-04-16T05:56:10.249 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:10.249 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24263,"name":"cephfs.smithi122.lojjdp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.122:6825/194115669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":194115669},{"type":"v1","addr":"172.21.15.122:6825","nonce":194115669}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":24273,"name":"cephfs.smithi122.rulewi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1078307975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1078307975},{"type":"v1","addr":"172.21.15.122:6827","nonce":1078307975}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34400,"name":"cephfs.smithi027.hkhpuq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/2831468256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2831468256},{"type":"v1","addr":"172.21.15.27:6827","nonce":2831468256}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T05:40:11.858650+0000","modified":"2024-04-16T05:40:32.701649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14462},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14462":{"gid":14462,"name":"cephfs.smithi027.ejoytk","rank":0,"incarnation":12,"state":"up:active","state_seq":5,"addr":"172.21.15.27:6829/4115272713","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":4115272713},{"type":"v1","addr":"172.21.15.27:6829","nonce":4115272713}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T05:56:10.249 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 18 2024-04-16T05:56:10.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.253+0000 7fb322400640 1 -- 172.21.15.27:0/1493250280 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb30006eb70 msgr2=0x7fb300071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:10.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.253+0000 7fb322400640 1 --2- 172.21.15.27:0/1493250280 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb30006eb70 0x7fb300071020 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb304007fd0 tx=0x7fb304009e90 comp rx=0 tx=0).stop 2024-04-16T05:56:10.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.253+0000 7fb322400640 1 -- 172.21.15.27:0/1493250280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31c0fd000 msgr2=0x7fb31c114d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:10.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.253+0000 7fb322400640 1 --2- 172.21.15.27:0/1493250280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31c0fd000 0x7fb31c114d50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb30c004930 tx=0x7fb30c009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:10.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.253+0000 7fb322400640 1 -- 172.21.15.27:0/1493250280 shutdown_connections 2024-04-16T05:56:10.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.253+0000 7fb322400640 1 --2- 172.21.15.27:0/1493250280 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fb30006eb70 0x7fb300071020 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:10.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.253+0000 7fb322400640 1 --2- 172.21.15.27:0/1493250280 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb31c0fd9a0 0x7fb31c115290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:10.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.253+0000 7fb322400640 1 --2- 172.21.15.27:0/1493250280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb31c0fd000 0x7fb31c114d50 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:10.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.253+0000 7fb322400640 1 -- 172.21.15.27:0/1493250280 >> 172.21.15.27:0/1493250280 conn(0x7fb31c0f6a90 msgr2=0x7fb31c0f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:10.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.253+0000 7fb322400640 1 -- 172.21.15.27:0/1493250280 shutdown_connections 2024-04-16T05:56:10.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.253+0000 7fb322400640 1 -- 172.21.15.27:0/1493250280 wait complete. 2024-04-16T05:56:10.269 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 19 2024-04-16T05:56:10.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.496+0000 7f543e400640 1 -- 172.21.15.27:0/3589737195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54381029b0 msgr2=0x7f54380fc040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:10.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.496+0000 7f543e400640 1 --2- 172.21.15.27:0/3589737195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54381029b0 0x7f54380fc040 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f54200048d0 tx=0x7f5420009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:10.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.496+0000 7f543e400640 1 -- 172.21.15.27:0/3589737195 shutdown_connections 2024-04-16T05:56:10.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.496+0000 7f543e400640 1 --2- 172.21.15.27:0/3589737195 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54381029b0 0x7f54380fc040 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:10.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.496+0000 7f543e400640 1 --2- 172.21.15.27:0/3589737195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5438102010 0x7f54381023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:10.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.496+0000 7f543e400640 1 -- 172.21.15.27:0/3589737195 >> 172.21.15.27:0/3589737195 conn(0x7f54380f7b80 msgr2=0x7f54380f9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:10.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.496+0000 7f543e400640 1 -- 172.21.15.27:0/3589737195 shutdown_connections 2024-04-16T05:56:10.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.496+0000 7f543e400640 1 -- 172.21.15.27:0/3589737195 wait complete. 2024-04-16T05:56:10.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.497+0000 7f543e400640 1 Processor -- start 2024-04-16T05:56:10.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.497+0000 7f543e400640 1 -- start start 2024-04-16T05:56:10.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.497+0000 7f543e400640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5438102010 0x7f543806f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:10.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.498+0000 7f543e400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54381029b0 0x7f543806fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:10.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.498+0000 7f543e400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54380716b0 con 0x7f54381029b0 2024-04-16T05:56:10.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.498+0000 7f543e400640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5438070250 con 0x7f5438102010 2024-04-16T05:56:10.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.498+0000 7f5437e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54381029b0 0x7f543806fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:10.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.498+0000 7f5437e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54381029b0 0x7f543806fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34058/0 (socket says 172.21.15.27:34058) 2024-04-16T05:56:10.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.498+0000 7f5437e00640 1 -- 172.21.15.27:0/1050957009 learned_addr learned my addr 172.21.15.27:0/1050957009 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:10.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.498+0000 7f543d000640 1 --2- 172.21.15.27:0/1050957009 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5438102010 0x7f543806f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:10.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.498+0000 7f5437e00640 1 -- 172.21.15.27:0/1050957009 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5438102010 msgr2=0x7f543806f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:10.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.498+0000 7f5437e00640 1 --2- 172.21.15.27:0/1050957009 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5438102010 0x7f543806f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:10.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.498+0000 7f5437e00640 1 -- 172.21.15.27:0/1050957009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5420005040 con 0x7f54381029b0 2024-04-16T05:56:10.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.499+0000 7f5437e00640 1 --2- 172.21.15.27:0/1050957009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54381029b0 0x7f543806fd10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f5420004930 tx=0x7f5420009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:10.498 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.499+0000 7f5434c00640 1 -- 172.21.15.27:0/1050957009 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54200094c0 con 0x7f54381029b0 2024-04-16T05:56:10.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.499+0000 7f5434c00640 1 -- 172.21.15.27:0/1050957009 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5420032e50 con 0x7f54381029b0 2024-04-16T05:56:10.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.499+0000 7f543e400640 1 -- 172.21.15.27:0/1050957009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54380704d0 con 0x7f54381029b0 2024-04-16T05:56:10.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.499+0000 7f5434c00640 1 -- 172.21.15.27:0/1050957009 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54200433b0 con 0x7f54381029b0 2024-04-16T05:56:10.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.499+0000 7f543e400640 1 -- 172.21.15.27:0/1050957009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54381191c0 con 0x7f54381029b0 2024-04-16T05:56:10.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.500+0000 7f543e400640 1 -- 172.21.15.27:0/1050957009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54380fd780 con 0x7f54381029b0 2024-04-16T05:56:10.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.501+0000 7f5434c00640 1 -- 172.21.15.27:0/1050957009 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f542002f070 con 0x7f54381029b0 2024-04-16T05:56:10.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.502+0000 7f5434c00640 1 --2- 172.21.15.27:0/1050957009 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f541c06eb70 0x7f541c071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:10.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.503+0000 7f5434c00640 1 -- 172.21.15.27:0/1050957009 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f54200b09c0 con 0x7f54381029b0 2024-04-16T05:56:10.501 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.503+0000 7f543d000640 1 --2- 172.21.15.27:0/1050957009 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f541c06eb70 0x7f541c071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:10.502 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.503+0000 7f543d000640 1 --2- 172.21.15.27:0/1050957009 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f541c06eb70 0x7f541c071020 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5428004ce0 tx=0x7f542800b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:10.506 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.507+0000 7f5434c00640 1 -- 172.21.15.27:0/1050957009 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f54200794d0 con 0x7f54381029b0 2024-04-16T05:56:10.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.758+0000 7f543e400640 1 -- 172.21.15.27:0/1050957009 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 19, "format": "json"} v 0) v1 -- 0x7f54381091e0 con 0x7f54381029b0 2024-04-16T05:56:10.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.759+0000 7f5434c00640 1 -- 172.21.15.27:0/1050957009 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 19, "format": "json"}]=0 dumped fsmap epoch 19 v36) v1 ==== 107+0+4138 (secure 0 0 0) 0x7f542007cb90 con 0x7f54381029b0 2024-04-16T05:56:10.759 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:10.759 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24263,"name":"cephfs.smithi122.lojjdp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":4,"addr":"172.21.15.122:6825/194115669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":194115669},{"type":"v1","addr":"172.21.15.122:6825","nonce":194115669}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15},{"gid":24273,"name":"cephfs.smithi122.rulewi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1078307975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1078307975},{"type":"v1","addr":"172.21.15.122:6827","nonce":1078307975}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34400,"name":"cephfs.smithi027.hkhpuq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/2831468256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2831468256},{"type":"v1","addr":"172.21.15.27:6827","nonce":2831468256}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T05:40:11.858650+0000","modified":"2024-04-16T05:50:45.882805+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T05:56:10.759 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 19 2024-04-16T05:56:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.762+0000 7f543e400640 1 -- 172.21.15.27:0/1050957009 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f541c06eb70 msgr2=0x7f541c071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.762+0000 7f543e400640 1 --2- 172.21.15.27:0/1050957009 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f541c06eb70 0x7f541c071020 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5428004ce0 tx=0x7f542800b040 comp rx=0 tx=0).stop 2024-04-16T05:56:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.762+0000 7f543e400640 1 -- 172.21.15.27:0/1050957009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54381029b0 msgr2=0x7f543806fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.762+0000 7f543e400640 1 --2- 172.21.15.27:0/1050957009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54381029b0 0x7f543806fd10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f5420004930 tx=0x7f5420009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.762+0000 7f543e400640 1 -- 172.21.15.27:0/1050957009 shutdown_connections 2024-04-16T05:56:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.762+0000 7f543e400640 1 --2- 172.21.15.27:0/1050957009 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f541c06eb70 0x7f541c071020 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.762+0000 7f543e400640 1 --2- 172.21.15.27:0/1050957009 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54381029b0 0x7f543806fd10 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.762+0000 7f543e400640 1 --2- 172.21.15.27:0/1050957009 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5438102010 0x7f543806f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.762+0000 7f543e400640 1 -- 172.21.15.27:0/1050957009 >> 172.21.15.27:0/1050957009 conn(0x7f54380f7b80 msgr2=0x7f54380f8b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.762+0000 7f543e400640 1 -- 172.21.15.27:0/1050957009 shutdown_connections 2024-04-16T05:56:10.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:10.762+0000 7f543e400640 1 -- 172.21.15.27:0/1050957009 wait complete. 2024-04-16T05:56:10.782 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 20 2024-04-16T05:56:11.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.011+0000 7ff7fec00640 1 -- 172.21.15.27:0/2553540022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8104bd0 msgr2=0x7ff7f80fe260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:11.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.011+0000 7ff7fec00640 1 --2- 172.21.15.27:0/2553540022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8104bd0 0x7ff7f80fe260 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff7e40048d0 tx=0x7ff7e4009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:11.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.011+0000 7ff7fec00640 1 -- 172.21.15.27:0/2553540022 shutdown_connections 2024-04-16T05:56:11.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.011+0000 7ff7fec00640 1 --2- 172.21.15.27:0/2553540022 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8104bd0 0x7ff7f80fe260 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:11.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.011+0000 7ff7fec00640 1 --2- 172.21.15.27:0/2553540022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7f8104230 0x7ff7f8104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:11.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.011+0000 7ff7fec00640 1 -- 172.21.15.27:0/2553540022 >> 172.21.15.27:0/2553540022 conn(0x7ff7f80f9fc0 msgr2=0x7ff7f80fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:11.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.011+0000 7ff7fec00640 1 -- 172.21.15.27:0/2553540022 shutdown_connections 2024-04-16T05:56:11.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.011+0000 7ff7fec00640 1 -- 172.21.15.27:0/2553540022 wait complete. 2024-04-16T05:56:11.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.012+0000 7ff7fec00640 1 Processor -- start 2024-04-16T05:56:11.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.012+0000 7ff7fec00640 1 -- start start 2024-04-16T05:56:11.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.013+0000 7ff7fec00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7f8104230 0x7ff7f8114d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:11.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.013+0000 7ff7fec00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8104bd0 0x7ff7f8115260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:11.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.013+0000 7ff7fec00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f8115860 con 0x7ff7f8104bd0 2024-04-16T05:56:11.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.013+0000 7ff7fec00640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f810eeb0 con 0x7ff7f8104230 2024-04-16T05:56:11.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.013+0000 7ff7fce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8104bd0 0x7ff7f8115260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:11.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.013+0000 7ff7fce00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8104bd0 0x7ff7f8115260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34080/0 (socket says 172.21.15.27:34080) 2024-04-16T05:56:11.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.013+0000 7ff7fce00640 1 -- 172.21.15.27:0/1084842202 learned_addr learned my addr 172.21.15.27:0/1084842202 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:11.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.014+0000 7ff7fd800640 1 --2- 172.21.15.27:0/1084842202 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7f8104230 0x7ff7f8114d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:11.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.014+0000 7ff7fce00640 1 -- 172.21.15.27:0/1084842202 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7f8104230 msgr2=0x7ff7f8114d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:11.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.014+0000 7ff7fce00640 1 --2- 172.21.15.27:0/1084842202 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7f8104230 0x7ff7f8114d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:11.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.014+0000 7ff7fce00640 1 -- 172.21.15.27:0/1084842202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e4005040 con 0x7ff7f8104bd0 2024-04-16T05:56:11.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.014+0000 7ff7fce00640 1 --2- 172.21.15.27:0/1084842202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8104bd0 0x7ff7f8115260 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff7e40047f0 tx=0x7ff7e4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:11.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.015+0000 7ff7f5600640 1 -- 172.21.15.27:0/1084842202 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7e40094c0 con 0x7ff7f8104bd0 2024-04-16T05:56:11.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.015+0000 7ff7f5600640 1 -- 172.21.15.27:0/1084842202 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff7e4032e50 con 0x7ff7f8104bd0 2024-04-16T05:56:11.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.015+0000 7ff7f5600640 1 -- 172.21.15.27:0/1084842202 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7e40433b0 con 0x7ff7f8104bd0 2024-04-16T05:56:11.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.015+0000 7ff7fec00640 1 -- 172.21.15.27:0/1084842202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7f810f130 con 0x7ff7f8104bd0 2024-04-16T05:56:11.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.015+0000 7ff7fec00640 1 -- 172.21.15.27:0/1084842202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f810f5f0 con 0x7ff7f8104bd0 2024-04-16T05:56:11.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.016+0000 7ff7fec00640 1 -- 172.21.15.27:0/1084842202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7f80ff9a0 con 0x7ff7f8104bd0 2024-04-16T05:56:11.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.017+0000 7ff7f5600640 1 -- 172.21.15.27:0/1084842202 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7ff7e402f070 con 0x7ff7f8104bd0 2024-04-16T05:56:11.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.019+0000 7ff7f5600640 1 --2- 172.21.15.27:0/1084842202 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff7e006eb70 0x7ff7e0071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:11.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.019+0000 7ff7f5600640 1 -- 172.21.15.27:0/1084842202 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7ff7e40b08e0 con 0x7ff7f8104bd0 2024-04-16T05:56:11.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.019+0000 7ff7fd800640 1 --2- 172.21.15.27:0/1084842202 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff7e006eb70 0x7ff7e0071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:11.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.020+0000 7ff7fd800640 1 --2- 172.21.15.27:0/1084842202 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff7e006eb70 0x7ff7e0071020 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec007fd0 tx=0x7ff7ec009e90 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:11.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:10 smithi122 ceph-mon[80930]: pgmap v362: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:11.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:10 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1856229583' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 17, "format": "json"}]: dispatch 2024-04-16T05:56:11.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:10 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1493250280' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 18, "format": "json"}]: dispatch 2024-04-16T05:56:11.021 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.023+0000 7ff7f5600640 1 -- 172.21.15.27:0/1084842202 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7ff7e4079490 con 0x7ff7f8104bd0 2024-04-16T05:56:11.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:10 smithi027 ceph-mon[93635]: pgmap v362: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:11.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:10 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1856229583' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 17, "format": "json"}]: dispatch 2024-04-16T05:56:11.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:10 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1493250280' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 18, "format": "json"}]: dispatch 2024-04-16T05:56:11.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.273+0000 7ff7fec00640 1 -- 172.21.15.27:0/1084842202 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 20, "format": "json"} v 0) v1 -- 0x7ff7f810b390 con 0x7ff7f8104bd0 2024-04-16T05:56:11.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.275+0000 7ff7f5600640 1 -- 172.21.15.27:0/1084842202 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 20, "format": "json"}]=0 dumped fsmap epoch 20 v36) v1 ==== 107+0+4149 (secure 0 0 0) 0x7ff7e407cb50 con 0x7ff7f8104bd0 2024-04-16T05:56:11.273 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:11.274 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24273,"name":"cephfs.smithi122.rulewi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1078307975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1078307975},{"type":"v1","addr":"172.21.15.122:6827","nonce":1078307975}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34400,"name":"cephfs.smithi027.hkhpuq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/2831468256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2831468256},{"type":"v1","addr":"172.21.15.27:6827","nonce":2831468256}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T05:40:11.858650+0000","modified":"2024-04-16T05:50:45.889399+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24263":{"gid":24263,"name":"cephfs.smithi122.lojjdp","rank":0,"incarnation":20,"state":"up:replay","state_seq":4,"addr":"172.21.15.122:6825/194115669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":194115669},{"type":"v1","addr":"172.21.15.122:6825","nonce":194115669}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T05:56:11.274 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 20 2024-04-16T05:56:11.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.278+0000 7ff7fec00640 1 -- 172.21.15.27:0/1084842202 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff7e006eb70 msgr2=0x7ff7e0071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:11.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.278+0000 7ff7fec00640 1 --2- 172.21.15.27:0/1084842202 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff7e006eb70 0x7ff7e0071020 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec007fd0 tx=0x7ff7ec009e90 comp rx=0 tx=0).stop 2024-04-16T05:56:11.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.278+0000 7ff7fec00640 1 -- 172.21.15.27:0/1084842202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8104bd0 msgr2=0x7ff7f8115260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:11.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.278+0000 7ff7fec00640 1 --2- 172.21.15.27:0/1084842202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8104bd0 0x7ff7f8115260 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff7e40047f0 tx=0x7ff7e4009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:11.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.278+0000 7ff7fec00640 1 -- 172.21.15.27:0/1084842202 shutdown_connections 2024-04-16T05:56:11.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.278+0000 7ff7fec00640 1 --2- 172.21.15.27:0/1084842202 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7ff7e006eb70 0x7ff7e0071020 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:11.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.278+0000 7ff7fec00640 1 --2- 172.21.15.27:0/1084842202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7f8104bd0 0x7ff7f8115260 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:11.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.278+0000 7ff7fec00640 1 --2- 172.21.15.27:0/1084842202 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7f8104230 0x7ff7f8114d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:11.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.278+0000 7ff7fec00640 1 -- 172.21.15.27:0/1084842202 >> 172.21.15.27:0/1084842202 conn(0x7ff7f80f9fc0 msgr2=0x7ff7f80fbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:11.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.278+0000 7ff7fec00640 1 -- 172.21.15.27:0/1084842202 shutdown_connections 2024-04-16T05:56:11.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.278+0000 7ff7fec00640 1 -- 172.21.15.27:0/1084842202 wait complete. 2024-04-16T05:56:11.298 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 21 2024-04-16T05:56:11.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.531+0000 7f16c3200640 1 -- 172.21.15.27:0/952840894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16c4102040 msgr2=0x7f16c4102410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:11.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.531+0000 7f16c3200640 1 --2- 172.21.15.27:0/952840894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16c4102040 0x7f16c4102410 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f16b40048d0 tx=0x7f16b4009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:11.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.531+0000 7f16c3200640 1 -- 172.21.15.27:0/952840894 shutdown_connections 2024-04-16T05:56:11.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.531+0000 7f16c3200640 1 --2- 172.21.15.27:0/952840894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16c41029e0 0x7f16c4064380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:11.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.531+0000 7f16c3200640 1 --2- 172.21.15.27:0/952840894 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16c4102040 0x7f16c4102410 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:11.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.531+0000 7f16c3200640 1 -- 172.21.15.27:0/952840894 >> 172.21.15.27:0/952840894 conn(0x7f16c4071010 msgr2=0x7f16c4071410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:11.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.531+0000 7f16c3200640 1 -- 172.21.15.27:0/952840894 shutdown_connections 2024-04-16T05:56:11.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.531+0000 7f16c3200640 1 -- 172.21.15.27:0/952840894 wait complete. 2024-04-16T05:56:11.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.532+0000 7f16c3200640 1 Processor -- start 2024-04-16T05:56:11.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.532+0000 7f16c3200640 1 -- start start 2024-04-16T05:56:11.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.532+0000 7f16c3200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16c4102040 0x7f16c4114d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:11.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.532+0000 7f16c3200640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16c41029e0 0x7f16c41152c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:11.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.532+0000 7f16c3200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c41158c0 con 0x7f16c4102040 2024-04-16T05:56:11.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.532+0000 7f16c3200640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c410ef10 con 0x7f16c41029e0 2024-04-16T05:56:11.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.532+0000 7f16c1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16c4102040 0x7f16c4114d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:11.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.533+0000 7f16c1e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16c4102040 0x7f16c4114d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34104/0 (socket says 172.21.15.27:34104) 2024-04-16T05:56:11.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.533+0000 7f16c1e00640 1 -- 172.21.15.27:0/858596086 learned_addr learned my addr 172.21.15.27:0/858596086 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:11.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.533+0000 7f16c1e00640 1 -- 172.21.15.27:0/858596086 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16c41029e0 msgr2=0x7f16c41152c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:11.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.533+0000 7f16c1e00640 1 --2- 172.21.15.27:0/858596086 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16c41029e0 0x7f16c41152c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:11.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.533+0000 7f16c1e00640 1 -- 172.21.15.27:0/858596086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b4005040 con 0x7f16c4102040 2024-04-16T05:56:11.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.533+0000 7f16c1e00640 1 --2- 172.21.15.27:0/858596086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16c4102040 0x7f16c4114d80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f16b40047f0 tx=0x7f16b4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:11.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.533+0000 7f16ba000640 1 -- 172.21.15.27:0/858596086 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16b40094c0 con 0x7f16c4102040 2024-04-16T05:56:11.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.534+0000 7f16ba000640 1 -- 172.21.15.27:0/858596086 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f16b4032e50 con 0x7f16c4102040 2024-04-16T05:56:11.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.534+0000 7f16ba000640 1 -- 172.21.15.27:0/858596086 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16b40433b0 con 0x7f16c4102040 2024-04-16T05:56:11.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.534+0000 7f16c3200640 1 -- 172.21.15.27:0/858596086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c410f190 con 0x7f16c4102040 2024-04-16T05:56:11.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.534+0000 7f16c3200640 1 -- 172.21.15.27:0/858596086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c410f650 con 0x7f16c4102040 2024-04-16T05:56:11.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.535+0000 7f16c3200640 1 -- 172.21.15.27:0/858596086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c40fd7e0 con 0x7f16c4102040 2024-04-16T05:56:11.535 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.536+0000 7f16ba000640 1 -- 172.21.15.27:0/858596086 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f16b402f070 con 0x7f16c4102040 2024-04-16T05:56:11.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.537+0000 7f16ba000640 1 --2- 172.21.15.27:0/858596086 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f16a406eb70 0x7f16a4071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:11.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.537+0000 7f16ba000640 1 -- 172.21.15.27:0/858596086 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f16b40b0890 con 0x7f16c4102040 2024-04-16T05:56:11.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.537+0000 7f16c1400640 1 --2- 172.21.15.27:0/858596086 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f16a406eb70 0x7f16a4071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:11.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.538+0000 7f16c1400640 1 --2- 172.21.15.27:0/858596086 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f16a406eb70 0x7f16a4071020 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f16c4110640 tx=0x7f16ac004f70 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:11.541 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.542+0000 7f16ba000640 1 -- 172.21.15.27:0/858596086 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f16b4079440 con 0x7f16c4102040 2024-04-16T05:56:11.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.790+0000 7f16c3200640 1 -- 172.21.15.27:0/858596086 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 21, "format": "json"} v 0) v1 -- 0x7f16c404bcc0 con 0x7f16c4102040 2024-04-16T05:56:11.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.791+0000 7f16ba000640 1 -- 172.21.15.27:0/858596086 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 21, "format": "json"}]=0 dumped fsmap epoch 21 v36) v1 ==== 107+0+4154 (secure 0 0 0) 0x7f16b407cb00 con 0x7f16c4102040 2024-04-16T05:56:11.790 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:11.790 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24273,"name":"cephfs.smithi122.rulewi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1078307975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1078307975},{"type":"v1","addr":"172.21.15.122:6827","nonce":1078307975}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34400,"name":"cephfs.smithi027.hkhpuq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/2831468256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2831468256},{"type":"v1","addr":"172.21.15.27:6827","nonce":2831468256}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T05:40:11.858650+0000","modified":"2024-04-16T05:50:45.910866+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24263":{"gid":24263,"name":"cephfs.smithi122.lojjdp","rank":0,"incarnation":20,"state":"up:reconnect","state_seq":158,"addr":"172.21.15.122:6825/194115669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":194115669},{"type":"v1","addr":"172.21.15.122:6825","nonce":194115669}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T05:56:11.791 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 21 2024-04-16T05:56:11.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.794+0000 7f16c3200640 1 -- 172.21.15.27:0/858596086 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f16a406eb70 msgr2=0x7f16a4071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:11.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.794+0000 7f16c3200640 1 --2- 172.21.15.27:0/858596086 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f16a406eb70 0x7f16a4071020 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f16c4110640 tx=0x7f16ac004f70 comp rx=0 tx=0).stop 2024-04-16T05:56:11.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.794+0000 7f16c3200640 1 -- 172.21.15.27:0/858596086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16c4102040 msgr2=0x7f16c4114d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:11.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.794+0000 7f16c3200640 1 --2- 172.21.15.27:0/858596086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16c4102040 0x7f16c4114d80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f16b40047f0 tx=0x7f16b4009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:11.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.794+0000 7f16c3200640 1 -- 172.21.15.27:0/858596086 shutdown_connections 2024-04-16T05:56:11.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.794+0000 7f16c3200640 1 --2- 172.21.15.27:0/858596086 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f16a406eb70 0x7f16a4071020 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:11.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.794+0000 7f16c3200640 1 --2- 172.21.15.27:0/858596086 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16c41029e0 0x7f16c41152c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:11.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.794+0000 7f16c3200640 1 --2- 172.21.15.27:0/858596086 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16c4102040 0x7f16c4114d80 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:11.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.794+0000 7f16c3200640 1 -- 172.21.15.27:0/858596086 >> 172.21.15.27:0/858596086 conn(0x7f16c4071010 msgr2=0x7f16c40fa170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:11.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.794+0000 7f16c3200640 1 -- 172.21.15.27:0/858596086 shutdown_connections 2024-04-16T05:56:11.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:11.794+0000 7f16c3200640 1 -- 172.21.15.27:0/858596086 wait complete. 2024-04-16T05:56:11.812 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 22 2024-04-16T05:56:12.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:11 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1050957009' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-16T05:56:12.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:11 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1084842202' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-16T05:56:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.029+0000 7f780be00640 1 -- 172.21.15.27:0/9072393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f780c104210 msgr2=0x7f780c1045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.029+0000 7f780be00640 1 --2- 172.21.15.27:0/9072393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f780c104210 0x7f780c1045e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f77fc0048d0 tx=0x7f77fc009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.030+0000 7f780be00640 1 -- 172.21.15.27:0/9072393 shutdown_connections 2024-04-16T05:56:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.030+0000 7f780be00640 1 --2- 172.21.15.27:0/9072393 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f780c104bb0 0x7f780c0fe240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.030+0000 7f780be00640 1 --2- 172.21.15.27:0/9072393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f780c104210 0x7f780c1045e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.030+0000 7f780be00640 1 -- 172.21.15.27:0/9072393 >> 172.21.15.27:0/9072393 conn(0x7f780c0f9fc0 msgr2=0x7f780c0fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.030+0000 7f780be00640 1 -- 172.21.15.27:0/9072393 shutdown_connections 2024-04-16T05:56:12.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.030+0000 7f780be00640 1 -- 172.21.15.27:0/9072393 wait complete. 2024-04-16T05:56:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.031+0000 7f780be00640 1 Processor -- start 2024-04-16T05:56:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.031+0000 7f780be00640 1 -- start start 2024-04-16T05:56:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.031+0000 7f780be00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f780c104210 0x7f780c114d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.031+0000 7f780be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f780c104bb0 0x7f780c115260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.031+0000 7f780be00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f780c115860 con 0x7f780c104bb0 2024-04-16T05:56:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.031+0000 7f780be00640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f780c10eeb0 con 0x7f780c104210 2024-04-16T05:56:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.032+0000 7f780a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f780c104bb0 0x7f780c115260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.032+0000 7f780a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f780c104bb0 0x7f780c115260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34122/0 (socket says 172.21.15.27:34122) 2024-04-16T05:56:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.032+0000 7f780a000640 1 -- 172.21.15.27:0/3608287158 learned_addr learned my addr 172.21.15.27:0/3608287158 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.032+0000 7f780aa00640 1 --2- 172.21.15.27:0/3608287158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f780c104210 0x7f780c114d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.032+0000 7f780a000640 1 -- 172.21.15.27:0/3608287158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f780c104210 msgr2=0x7f780c114d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.032+0000 7f780a000640 1 --2- 172.21.15.27:0/3608287158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f780c104210 0x7f780c114d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:12.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.032+0000 7f780a000640 1 -- 172.21.15.27:0/3608287158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77fc005040 con 0x7f780c104bb0 2024-04-16T05:56:12.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.032+0000 7f780aa00640 1 --2- 172.21.15.27:0/3608287158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f780c104210 0x7f780c114d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:56:12.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.033+0000 7f780a000640 1 --2- 172.21.15.27:0/3608287158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f780c104bb0 0x7f780c115260 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f77f800eb70 tx=0x7f77f800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:12.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.033+0000 7f7802a00640 1 -- 172.21.15.27:0/3608287158 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77f8007c30 con 0x7f780c104bb0 2024-04-16T05:56:12.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.033+0000 7f780be00640 1 -- 172.21.15.27:0/3608287158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f780c10f130 con 0x7f780c104bb0 2024-04-16T05:56:12.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.033+0000 7f7802a00640 1 -- 172.21.15.27:0/3608287158 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f77f800f040 con 0x7f780c104bb0 2024-04-16T05:56:12.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.033+0000 7f7802a00640 1 -- 172.21.15.27:0/3608287158 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77f80095b0 con 0x7f780c104bb0 2024-04-16T05:56:12.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.033+0000 7f780be00640 1 -- 172.21.15.27:0/3608287158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f780c10f650 con 0x7f780c104bb0 2024-04-16T05:56:12.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.034+0000 7f780be00640 1 -- 172.21.15.27:0/3608287158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f780c0ff980 con 0x7f780c104bb0 2024-04-16T05:56:12.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.036+0000 7f7802a00640 1 -- 172.21.15.27:0/3608287158 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f77f8009730 con 0x7f780c104bb0 2024-04-16T05:56:12.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.037+0000 7f7802a00640 1 --2- 172.21.15.27:0/3608287158 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f77f006eb70 0x7f77f0071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:12.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.037+0000 7f7802a00640 1 -- 172.21.15.27:0/3608287158 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f77f808e7f0 con 0x7f780c104bb0 2024-04-16T05:56:12.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.037+0000 7f780aa00640 1 --2- 172.21.15.27:0/3608287158 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f77f006eb70 0x7f77f0071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:12.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.038+0000 7f780aa00640 1 --2- 172.21.15.27:0/3608287158 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f77f006eb70 0x7f77f0071020 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f77fc000c00 tx=0x7f77fc009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:12.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.042+0000 7f7802a00640 1 -- 172.21.15.27:0/3608287158 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f77f80573a0 con 0x7f780c104bb0 2024-04-16T05:56:12.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:11 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1050957009' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 19, "format": "json"}]: dispatch 2024-04-16T05:56:12.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:11 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1084842202' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 20, "format": "json"}]: dispatch 2024-04-16T05:56:12.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.290+0000 7f780be00640 1 -- 172.21.15.27:0/3608287158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 22, "format": "json"} v 0) v1 -- 0x7f780c10b3d0 con 0x7f780c104bb0 2024-04-16T05:56:12.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.291+0000 7f7802a00640 1 -- 172.21.15.27:0/3608287158 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 22, "format": "json"}]=0 dumped fsmap epoch 22 v36) v1 ==== 107+0+4151 (secure 0 0 0) 0x7f77f805aa60 con 0x7f780c104bb0 2024-04-16T05:56:12.290 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:12.290 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24273,"name":"cephfs.smithi122.rulewi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1078307975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1078307975},{"type":"v1","addr":"172.21.15.122:6827","nonce":1078307975}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34400,"name":"cephfs.smithi027.hkhpuq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/2831468256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2831468256},{"type":"v1","addr":"172.21.15.27:6827","nonce":2831468256}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T05:40:11.858650+0000","modified":"2024-04-16T05:50:46.905080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24263":{"gid":24263,"name":"cephfs.smithi122.lojjdp","rank":0,"incarnation":20,"state":"up:rejoin","state_seq":159,"addr":"172.21.15.122:6825/194115669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":194115669},{"type":"v1","addr":"172.21.15.122:6825","nonce":194115669}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T05:56:12.290 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 22 2024-04-16T05:56:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.294+0000 7f780be00640 1 -- 172.21.15.27:0/3608287158 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f77f006eb70 msgr2=0x7f77f0071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.294+0000 7f780be00640 1 --2- 172.21.15.27:0/3608287158 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f77f006eb70 0x7f77f0071020 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f77fc000c00 tx=0x7f77fc009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.294+0000 7f780be00640 1 -- 172.21.15.27:0/3608287158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f780c104bb0 msgr2=0x7f780c115260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.294+0000 7f780be00640 1 --2- 172.21.15.27:0/3608287158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f780c104bb0 0x7f780c115260 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f77f800eb70 tx=0x7f77f800b040 comp rx=0 tx=0).stop 2024-04-16T05:56:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.294+0000 7f780be00640 1 -- 172.21.15.27:0/3608287158 shutdown_connections 2024-04-16T05:56:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.294+0000 7f780be00640 1 --2- 172.21.15.27:0/3608287158 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f77f006eb70 0x7f77f0071020 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.294+0000 7f780be00640 1 --2- 172.21.15.27:0/3608287158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f780c104bb0 0x7f780c115260 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.294+0000 7f780be00640 1 --2- 172.21.15.27:0/3608287158 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f780c104210 0x7f780c114d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.294+0000 7f780be00640 1 -- 172.21.15.27:0/3608287158 >> 172.21.15.27:0/3608287158 conn(0x7f780c0f9fc0 msgr2=0x7f780c0fbee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:12.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.294+0000 7f780be00640 1 -- 172.21.15.27:0/3608287158 shutdown_connections 2024-04-16T05:56:12.294 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.294+0000 7f780be00640 1 -- 172.21.15.27:0/3608287158 wait complete. 2024-04-16T05:56:12.314 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 23 2024-04-16T05:56:12.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.524+0000 7f46b8c00640 1 -- 172.21.15.27:0/2456022919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b41029e0 msgr2=0x7f46b4064380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:12.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.524+0000 7f46b8c00640 1 --2- 172.21.15.27:0/2456022919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b41029e0 0x7f46b4064380 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f46a00048d0 tx=0x7f46a0009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:12.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.524+0000 7f46b8c00640 1 -- 172.21.15.27:0/2456022919 shutdown_connections 2024-04-16T05:56:12.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.524+0000 7f46b8c00640 1 --2- 172.21.15.27:0/2456022919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b41029e0 0x7f46b4064380 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:12.523 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.524+0000 7f46b8c00640 1 --2- 172.21.15.27:0/2456022919 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46b4102040 0x7f46b4102410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:12.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.524+0000 7f46b8c00640 1 -- 172.21.15.27:0/2456022919 >> 172.21.15.27:0/2456022919 conn(0x7f46b4071010 msgr2=0x7f46b4071410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:12.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.524+0000 7f46b8c00640 1 -- 172.21.15.27:0/2456022919 shutdown_connections 2024-04-16T05:56:12.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.524+0000 7f46b8c00640 1 -- 172.21.15.27:0/2456022919 wait complete. 2024-04-16T05:56:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.525+0000 7f46b8c00640 1 Processor -- start 2024-04-16T05:56:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.525+0000 7f46b8c00640 1 -- start start 2024-04-16T05:56:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.526+0000 7f46b8c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4102040 0x7f46b40fb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.526+0000 7f46b8c00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46b41029e0 0x7f46b40fb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.526+0000 7f46b8c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b40fcff0 con 0x7f46b4102040 2024-04-16T05:56:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.526+0000 7f46b8c00640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b40fbb00 con 0x7f46b41029e0 2024-04-16T05:56:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.526+0000 7f46b3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4102040 0x7f46b40fb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.526+0000 7f46b3400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4102040 0x7f46b40fb080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34146/0 (socket says 172.21.15.27:34146) 2024-04-16T05:56:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.526+0000 7f46b3400640 1 -- 172.21.15.27:0/360542721 learned_addr learned my addr 172.21.15.27:0/360542721 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.526+0000 7f46b2a00640 1 --2- 172.21.15.27:0/360542721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46b41029e0 0x7f46b40fb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:12.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.526+0000 7f46b3400640 1 -- 172.21.15.27:0/360542721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46b41029e0 msgr2=0x7f46b40fb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:12.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.527+0000 7f46b3400640 1 --2- 172.21.15.27:0/360542721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46b41029e0 0x7f46b40fb5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:12.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.527+0000 7f46b3400640 1 -- 172.21.15.27:0/360542721 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a0005040 con 0x7f46b4102040 2024-04-16T05:56:12.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.527+0000 7f46b3400640 1 --2- 172.21.15.27:0/360542721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4102040 0x7f46b40fb080 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f46a400b790 tx=0x7f46a40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:12.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.527+0000 7f46ab400640 1 -- 172.21.15.27:0/360542721 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46a400c700 con 0x7f46b4102040 2024-04-16T05:56:12.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.527+0000 7f46ab400640 1 -- 172.21.15.27:0/360542721 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f46a400cd30 con 0x7f46b4102040 2024-04-16T05:56:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.527+0000 7f46b8c00640 1 -- 172.21.15.27:0/360542721 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b40fbd80 con 0x7f46b4102040 2024-04-16T05:56:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.527+0000 7f46ab400640 1 -- 172.21.15.27:0/360542721 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f46a4013370 con 0x7f46b4102040 2024-04-16T05:56:12.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.527+0000 7f46b8c00640 1 -- 172.21.15.27:0/360542721 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b41193a0 con 0x7f46b4102040 2024-04-16T05:56:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.528+0000 7f46b8c00640 1 -- 172.21.15.27:0/360542721 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b40fd7e0 con 0x7f46b4102040 2024-04-16T05:56:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.529+0000 7f46ab400640 1 -- 172.21.15.27:0/360542721 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f46a4012070 con 0x7f46b4102040 2024-04-16T05:56:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.531+0000 7f46ab400640 1 --2- 172.21.15.27:0/360542721 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f469806eb70 0x7f4698071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.531+0000 7f46ab400640 1 -- 172.21.15.27:0/360542721 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f46a408e660 con 0x7f46b4102040 2024-04-16T05:56:12.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.531+0000 7f46b2a00640 1 --2- 172.21.15.27:0/360542721 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f469806eb70 0x7f4698071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:12.532 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.532+0000 7f46b2a00640 1 --2- 172.21.15.27:0/360542721 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f469806eb70 0x7f4698071020 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f46a002be70 tx=0x7f46a0009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:12.534 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.536+0000 7f46ab400640 1 -- 172.21.15.27:0/360542721 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f46a40571f0 con 0x7f46b4102040 2024-04-16T05:56:12.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.784+0000 7f46b8c00640 1 -- 172.21.15.27:0/360542721 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 23, "format": "json"} v 0) v1 -- 0x7f46b404bcc0 con 0x7f46b4102040 2024-04-16T05:56:12.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.786+0000 7f46ab400640 1 -- 172.21.15.27:0/360542721 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 23, "format": "json"}]=0 dumped fsmap epoch 23 v36) v1 ==== 107+0+4160 (secure 0 0 0) 0x7f46a405a8b0 con 0x7f46b4102040 2024-04-16T05:56:12.784 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:12.785 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24273,"name":"cephfs.smithi122.rulewi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1078307975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1078307975},{"type":"v1","addr":"172.21.15.122:6827","nonce":1078307975}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34400,"name":"cephfs.smithi027.hkhpuq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/2831468256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2831468256},{"type":"v1","addr":"172.21.15.27:6827","nonce":2831468256}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T05:40:11.858650+0000","modified":"2024-04-16T05:50:48.911395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24263":{"gid":24263,"name":"cephfs.smithi122.lojjdp","rank":0,"incarnation":20,"state":"up:active","state_seq":160,"addr":"172.21.15.122:6825/194115669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":194115669},{"type":"v1","addr":"172.21.15.122:6825","nonce":194115669}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24263,"qdb_cluster":[24263]},"id":1}]} 2024-04-16T05:56:12.785 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 23 2024-04-16T05:56:12.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.788+0000 7f46b8c00640 1 -- 172.21.15.27:0/360542721 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f469806eb70 msgr2=0x7f4698071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:12.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.788+0000 7f46b8c00640 1 --2- 172.21.15.27:0/360542721 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f469806eb70 0x7f4698071020 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f46a002be70 tx=0x7f46a0009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:12.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.788+0000 7f46b8c00640 1 -- 172.21.15.27:0/360542721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4102040 msgr2=0x7f46b40fb080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:12.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.788+0000 7f46b8c00640 1 --2- 172.21.15.27:0/360542721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4102040 0x7f46b40fb080 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f46a400b790 tx=0x7f46a40091d0 comp rx=0 tx=0).stop 2024-04-16T05:56:12.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.788+0000 7f46b8c00640 1 -- 172.21.15.27:0/360542721 shutdown_connections 2024-04-16T05:56:12.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.788+0000 7f46b8c00640 1 --2- 172.21.15.27:0/360542721 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f469806eb70 0x7f4698071020 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:12.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.788+0000 7f46b8c00640 1 --2- 172.21.15.27:0/360542721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46b41029e0 0x7f46b40fb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:12.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.788+0000 7f46b8c00640 1 --2- 172.21.15.27:0/360542721 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46b4102040 0x7f46b40fb080 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:12.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.788+0000 7f46b8c00640 1 -- 172.21.15.27:0/360542721 >> 172.21.15.27:0/360542721 conn(0x7f46b4071010 msgr2=0x7f46b40fa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:12.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.789+0000 7f46b8c00640 1 -- 172.21.15.27:0/360542721 shutdown_connections 2024-04-16T05:56:12.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:12.789+0000 7f46b8c00640 1 -- 172.21.15.27:0/360542721 wait complete. 2024-04-16T05:56:12.809 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 24 2024-04-16T05:56:13.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:12 smithi122 ceph-mon[80930]: pgmap v363: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:13.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:12 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/858596086' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-16T05:56:13.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:12 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/3608287158' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-16T05:56:13.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.026+0000 7f0a44000640 1 -- 172.21.15.27:0/2070901356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a3c104f70 msgr2=0x7f0a3c0fe630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.026+0000 7f0a44000640 1 --2- 172.21.15.27:0/2070901356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a3c104f70 0x7f0a3c0fe630 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0a280048d0 tx=0x7f0a28009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.026+0000 7f0a44000640 1 -- 172.21.15.27:0/2070901356 shutdown_connections 2024-04-16T05:56:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.026+0000 7f0a44000640 1 --2- 172.21.15.27:0/2070901356 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a3c104f70 0x7f0a3c0fe630 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.026+0000 7f0a44000640 1 --2- 172.21.15.27:0/2070901356 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a3c1045d0 0x7f0a3c1049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.026+0000 7f0a44000640 1 -- 172.21.15.27:0/2070901356 >> 172.21.15.27:0/2070901356 conn(0x7f0a3c0fa1b0 msgr2=0x7f0a3c0fc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.026+0000 7f0a44000640 1 -- 172.21.15.27:0/2070901356 shutdown_connections 2024-04-16T05:56:13.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.026+0000 7f0a44000640 1 -- 172.21.15.27:0/2070901356 wait complete. 2024-04-16T05:56:13.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.027+0000 7f0a44000640 1 Processor -- start 2024-04-16T05:56:13.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.027+0000 7f0a44000640 1 -- start start 2024-04-16T05:56:13.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.027+0000 7f0a44000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a3c1045d0 0x7f0a3c1150f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:13.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.027+0000 7f0a44000640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a3c104f70 0x7f0a3c115630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:13.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.027+0000 7f0a44000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3c115c30 con 0x7f0a3c1045d0 2024-04-16T05:56:13.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.027+0000 7f0a44000640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3c10f280 con 0x7f0a3c104f70 2024-04-16T05:56:13.027 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.027+0000 7f0a41600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a3c1045d0 0x7f0a3c1150f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:13.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.027+0000 7f0a41600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a3c1045d0 0x7f0a3c1150f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34164/0 (socket says 172.21.15.27:34164) 2024-04-16T05:56:13.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.027+0000 7f0a41600640 1 -- 172.21.15.27:0/3819209446 learned_addr learned my addr 172.21.15.27:0/3819209446 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:13.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.028+0000 7f0a41600640 1 -- 172.21.15.27:0/3819209446 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a3c104f70 msgr2=0x7f0a3c115630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:13.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.028+0000 7f0a41600640 1 --2- 172.21.15.27:0/3819209446 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a3c104f70 0x7f0a3c115630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:13.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.028+0000 7f0a41600640 1 -- 172.21.15.27:0/3819209446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a28005040 con 0x7f0a3c1045d0 2024-04-16T05:56:13.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.028+0000 7f0a41600640 1 --2- 172.21.15.27:0/3819209446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a3c1045d0 0x7f0a3c1150f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0a30002a50 tx=0x7f0a30004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:13.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.028+0000 7f0a39600640 1 -- 172.21.15.27:0/3819209446 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a30007c30 con 0x7f0a3c1045d0 2024-04-16T05:56:13.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.028+0000 7f0a39600640 1 -- 172.21.15.27:0/3819209446 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a3000f040 con 0x7f0a3c1045d0 2024-04-16T05:56:13.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.028+0000 7f0a39600640 1 -- 172.21.15.27:0/3819209446 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0a3000dbe0 con 0x7f0a3c1045d0 2024-04-16T05:56:13.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.028+0000 7f0a44000640 1 -- 172.21.15.27:0/3819209446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a3c10f560 con 0x7f0a3c1045d0 2024-04-16T05:56:13.028 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.028+0000 7f0a44000640 1 -- 172.21.15.27:0/3819209446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a3c10fa30 con 0x7f0a3c1045d0 2024-04-16T05:56:13.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.030+0000 7f0a44000640 1 -- 172.21.15.27:0/3819209446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a3c0ffd50 con 0x7f0a3c1045d0 2024-04-16T05:56:13.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.031+0000 7f0a39600640 1 -- 172.21.15.27:0/3819209446 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f0a3000dd40 con 0x7f0a3c1045d0 2024-04-16T05:56:13.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.032+0000 7f0a39600640 1 --2- 172.21.15.27:0/3819209446 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f0a2406eb70 0x7f0a24071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:13.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.032+0000 7f0a39600640 1 -- 172.21.15.27:0/3819209446 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f0a3008eb00 con 0x7f0a3c1045d0 2024-04-16T05:56:13.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.032+0000 7f0a40c00640 1 --2- 172.21.15.27:0/3819209446 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f0a2406eb70 0x7f0a24071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:13.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.033+0000 7f0a40c00640 1 --2- 172.21.15.27:0/3819209446 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f0a2406eb70 0x7f0a24071020 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0a28000c00 tx=0x7f0a28009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:13.036 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.038+0000 7f0a39600640 1 -- 172.21.15.27:0/3819209446 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f0a30057610 con 0x7f0a3c1045d0 2024-04-16T05:56:13.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:12 smithi027 ceph-mon[93635]: pgmap v363: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:13.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:12 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/858596086' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 21, "format": "json"}]: dispatch 2024-04-16T05:56:13.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:12 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/3608287158' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 22, "format": "json"}]: dispatch 2024-04-16T05:56:13.286 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.287+0000 7f0a44000640 1 -- 172.21.15.27:0/3819209446 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 24, "format": "json"} v 0) v1 -- 0x7f0a3c10b760 con 0x7f0a3c1045d0 2024-04-16T05:56:13.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.288+0000 7f0a39600640 1 -- 172.21.15.27:0/3819209446 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 24, "format": "json"}]=0 dumped fsmap epoch 24 v36) v1 ==== 107+0+5007 (secure 0 0 0) 0x7f0a3005acd0 con 0x7f0a3c1045d0 2024-04-16T05:56:13.287 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:13.287 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24273,"name":"cephfs.smithi122.rulewi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1078307975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1078307975},{"type":"v1","addr":"172.21.15.122:6827","nonce":1078307975}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34400,"name":"cephfs.smithi027.hkhpuq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/2831468256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2831468256},{"type":"v1","addr":"172.21.15.27:6827","nonce":2831468256}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":34432,"name":"cephfs.smithi027.ejoytk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6829/3219609959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3219609959},{"type":"v1","addr":"172.21.15.27:6829","nonce":3219609959}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T05:40:11.858650+0000","modified":"2024-04-16T05:50:48.911395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24263":{"gid":24263,"name":"cephfs.smithi122.lojjdp","rank":0,"incarnation":20,"state":"up:active","state_seq":160,"addr":"172.21.15.122:6825/194115669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":194115669},{"type":"v1","addr":"172.21.15.122:6825","nonce":194115669}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24263,"qdb_cluster":[24263]},"id":1}]} 2024-04-16T05:56:13.288 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 24 2024-04-16T05:56:13.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.291+0000 7f0a44000640 1 -- 172.21.15.27:0/3819209446 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f0a2406eb70 msgr2=0x7f0a24071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:13.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.291+0000 7f0a44000640 1 --2- 172.21.15.27:0/3819209446 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f0a2406eb70 0x7f0a24071020 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0a28000c00 tx=0x7f0a28009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:13.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.291+0000 7f0a44000640 1 -- 172.21.15.27:0/3819209446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a3c1045d0 msgr2=0x7f0a3c1150f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:13.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.291+0000 7f0a44000640 1 --2- 172.21.15.27:0/3819209446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a3c1045d0 0x7f0a3c1150f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0a30002a50 tx=0x7f0a30004600 comp rx=0 tx=0).stop 2024-04-16T05:56:13.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.291+0000 7f0a44000640 1 -- 172.21.15.27:0/3819209446 shutdown_connections 2024-04-16T05:56:13.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.291+0000 7f0a44000640 1 --2- 172.21.15.27:0/3819209446 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f0a2406eb70 0x7f0a24071020 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:13.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.291+0000 7f0a44000640 1 --2- 172.21.15.27:0/3819209446 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0a3c104f70 0x7f0a3c115630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:13.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.291+0000 7f0a44000640 1 --2- 172.21.15.27:0/3819209446 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0a3c1045d0 0x7f0a3c1150f0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:13.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.291+0000 7f0a44000640 1 -- 172.21.15.27:0/3819209446 >> 172.21.15.27:0/3819209446 conn(0x7f0a3c0fa1b0 msgr2=0x7f0a3c0fb0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:13.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.291+0000 7f0a44000640 1 -- 172.21.15.27:0/3819209446 shutdown_connections 2024-04-16T05:56:13.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.291+0000 7f0a44000640 1 -- 172.21.15.27:0/3819209446 wait complete. 2024-04-16T05:56:13.307 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 25 2024-04-16T05:56:13.521 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.522+0000 7fc274e00640 1 -- 172.21.15.27:0/3422469473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2700fcfd0 msgr2=0x7fc2700fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.522+0000 7fc274e00640 1 --2- 172.21.15.27:0/3422469473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2700fcfd0 0x7fc2700fd3a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc260004900 tx=0x7fc260009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.523+0000 7fc274e00640 1 -- 172.21.15.27:0/3422469473 shutdown_connections 2024-04-16T05:56:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.523+0000 7fc274e00640 1 --2- 172.21.15.27:0/3422469473 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2700fd970 0x7fc270106040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.523+0000 7fc274e00640 1 --2- 172.21.15.27:0/3422469473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2700fcfd0 0x7fc2700fd3a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.523+0000 7fc274e00640 1 -- 172.21.15.27:0/3422469473 >> 172.21.15.27:0/3422469473 conn(0x7fc2700f6ad0 msgr2=0x7fc2700f8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.523+0000 7fc274e00640 1 -- 172.21.15.27:0/3422469473 shutdown_connections 2024-04-16T05:56:13.522 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.523+0000 7fc274e00640 1 -- 172.21.15.27:0/3422469473 wait complete. 2024-04-16T05:56:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.524+0000 7fc274e00640 1 Processor -- start 2024-04-16T05:56:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.524+0000 7fc274e00640 1 -- start start 2024-04-16T05:56:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.524+0000 7fc274e00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2700fcfd0 0x7fc2700fae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.524+0000 7fc274e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2700fd970 0x7fc2700fb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.524+0000 7fc274e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2700fce00 con 0x7fc2700fd970 2024-04-16T05:56:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.524+0000 7fc274e00640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2700fb910 con 0x7fc2700fcfd0 2024-04-16T05:56:13.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.524+0000 7fc267e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2700fd970 0x7fc2700fb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.524+0000 7fc267e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2700fd970 0x7fc2700fb3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34190/0 (socket says 172.21.15.27:34190) 2024-04-16T05:56:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.524+0000 7fc267e00640 1 -- 172.21.15.27:0/1249866857 learned_addr learned my addr 172.21.15.27:0/1249866857 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.525+0000 7fc26f400640 1 --2- 172.21.15.27:0/1249866857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2700fcfd0 0x7fc2700fae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.525+0000 7fc267e00640 1 -- 172.21.15.27:0/1249866857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2700fcfd0 msgr2=0x7fc2700fae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.525+0000 7fc267e00640 1 --2- 172.21.15.27:0/1249866857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2700fcfd0 0x7fc2700fae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.525+0000 7fc267e00640 1 -- 172.21.15.27:0/1249866857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc260005040 con 0x7fc2700fd970 2024-04-16T05:56:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.525+0000 7fc26f400640 1 --2- 172.21.15.27:0/1249866857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2700fcfd0 0x7fc2700fae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:56:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.525+0000 7fc267e00640 1 --2- 172.21.15.27:0/1249866857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2700fd970 0x7fc2700fb3d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc25800b790 tx=0x7fc2580091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.525+0000 7fc267400640 1 -- 172.21.15.27:0/1249866857 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc25800c700 con 0x7fc2700fd970 2024-04-16T05:56:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.525+0000 7fc267400640 1 -- 172.21.15.27:0/1249866857 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc25800cd30 con 0x7fc2700fd970 2024-04-16T05:56:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.525+0000 7fc267400640 1 -- 172.21.15.27:0/1249866857 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc258013370 con 0x7fc2700fd970 2024-04-16T05:56:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.525+0000 7fc274e00640 1 -- 172.21.15.27:0/1249866857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2700fbb90 con 0x7fc2700fd970 2024-04-16T05:56:13.525 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.526+0000 7fc274e00640 1 -- 172.21.15.27:0/1249866857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc270119380 con 0x7fc2700fd970 2024-04-16T05:56:13.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.527+0000 7fc274e00640 1 -- 172.21.15.27:0/1249866857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc270065d80 con 0x7fc2700fd970 2024-04-16T05:56:13.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.528+0000 7fc267400640 1 -- 172.21.15.27:0/1249866857 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc258012070 con 0x7fc2700fd970 2024-04-16T05:56:13.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.529+0000 7fc267400640 1 --2- 172.21.15.27:0/1249866857 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc25406eb70 0x7fc254071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:13.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.529+0000 7fc267400640 1 -- 172.21.15.27:0/1249866857 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7fc25808e660 con 0x7fc2700fd970 2024-04-16T05:56:13.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.529+0000 7fc26f400640 1 --2- 172.21.15.27:0/1249866857 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc25406eb70 0x7fc254071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:13.528 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.530+0000 7fc26f400640 1 --2- 172.21.15.27:0/1249866857 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc25406eb70 0x7fc254071020 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc260009a30 tx=0x7fc260009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:13.536 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.537+0000 7fc267400640 1 -- 172.21.15.27:0/1249866857 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc2580571f0 con 0x7fc2700fd970 2024-04-16T05:56:13.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.786+0000 7fc274e00640 1 -- 172.21.15.27:0/1249866857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 25, "format": "json"} v 0) v1 -- 0x7fc27004bcc0 con 0x7fc2700fd970 2024-04-16T05:56:13.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.786+0000 7fc267400640 1 -- 172.21.15.27:0/1249866857 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 25, "format": "json"}]=0 dumped fsmap epoch 25 v36) v1 ==== 107+0+4206 (secure 0 0 0) 0x7fc25805a8b0 con 0x7fc2700fd970 2024-04-16T05:56:13.784 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:13.785 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24273,"name":"cephfs.smithi122.rulewi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.122:6827/1078307975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1078307975},{"type":"v1","addr":"172.21.15.122:6827","nonce":1078307975}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":34400,"name":"cephfs.smithi027.hkhpuq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/2831468256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2831468256},{"type":"v1","addr":"172.21.15.27:6827","nonce":2831468256}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":34432,"name":"cephfs.smithi027.ejoytk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6829/3219609959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3219609959},{"type":"v1","addr":"172.21.15.27:6829","nonce":3219609959}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T05:40:11.858650+0000","modified":"2024-04-16T05:51:15.646986+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T05:56:13.785 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 25 2024-04-16T05:56:13.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.788+0000 7fc274e00640 1 -- 172.21.15.27:0/1249866857 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc25406eb70 msgr2=0x7fc254071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:13.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.788+0000 7fc274e00640 1 --2- 172.21.15.27:0/1249866857 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc25406eb70 0x7fc254071020 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc260009a30 tx=0x7fc260009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:13.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.788+0000 7fc274e00640 1 -- 172.21.15.27:0/1249866857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2700fd970 msgr2=0x7fc2700fb3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:13.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.788+0000 7fc274e00640 1 --2- 172.21.15.27:0/1249866857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2700fd970 0x7fc2700fb3d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc25800b790 tx=0x7fc2580091d0 comp rx=0 tx=0).stop 2024-04-16T05:56:13.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.789+0000 7fc274e00640 1 -- 172.21.15.27:0/1249866857 shutdown_connections 2024-04-16T05:56:13.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.789+0000 7fc274e00640 1 --2- 172.21.15.27:0/1249866857 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc25406eb70 0x7fc254071020 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:13.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.789+0000 7fc274e00640 1 --2- 172.21.15.27:0/1249866857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2700fd970 0x7fc2700fb3d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:13.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.789+0000 7fc274e00640 1 --2- 172.21.15.27:0/1249866857 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2700fcfd0 0x7fc2700fae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:13.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.789+0000 7fc274e00640 1 -- 172.21.15.27:0/1249866857 >> 172.21.15.27:0/1249866857 conn(0x7fc2700f6ad0 msgr2=0x7fc2700f7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:13.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.789+0000 7fc274e00640 1 -- 172.21.15.27:0/1249866857 shutdown_connections 2024-04-16T05:56:13.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:13.789+0000 7fc274e00640 1 -- 172.21.15.27:0/1249866857 wait complete. 2024-04-16T05:56:13.805 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 26 2024-04-16T05:56:14.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:13 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/360542721' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-16T05:56:14.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:13 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/3819209446' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-16T05:56:14.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.031+0000 7f6763200640 1 -- 172.21.15.27:0/3475095224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67641029e0 msgr2=0x7f6764064380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:14.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.031+0000 7f6763200640 1 --2- 172.21.15.27:0/3475095224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67641029e0 0x7f6764064380 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f67500048d0 tx=0x7f6750009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:14.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.031+0000 7f6763200640 1 -- 172.21.15.27:0/3475095224 shutdown_connections 2024-04-16T05:56:14.029 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.031+0000 7f6763200640 1 --2- 172.21.15.27:0/3475095224 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67641029e0 0x7f6764064380 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:14.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.031+0000 7f6763200640 1 --2- 172.21.15.27:0/3475095224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6764102040 0x7f6764102410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:14.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.031+0000 7f6763200640 1 -- 172.21.15.27:0/3475095224 >> 172.21.15.27:0/3475095224 conn(0x7f6764071010 msgr2=0x7f6764071410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:14.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.031+0000 7f6763200640 1 -- 172.21.15.27:0/3475095224 shutdown_connections 2024-04-16T05:56:14.030 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.031+0000 7f6763200640 1 -- 172.21.15.27:0/3475095224 wait complete. 2024-04-16T05:56:14.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.032+0000 7f6763200640 1 Processor -- start 2024-04-16T05:56:14.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.032+0000 7f6763200640 1 -- start start 2024-04-16T05:56:14.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.033+0000 7f6763200640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6764102040 0x7f6764112b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:14.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.033+0000 7f6763200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67641029e0 0x7f67641130c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:14.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.033+0000 7f6763200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67641136c0 con 0x7f67641029e0 2024-04-16T05:56:14.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.033+0000 7f6763200640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676410cd10 con 0x7f6764102040 2024-04-16T05:56:14.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.033+0000 7f6761400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67641029e0 0x7f67641130c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:14.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.033+0000 7f6761e00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6764102040 0x7f6764112b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:14.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.034+0000 7f6761e00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6764102040 0x7f6764112b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.27:60778/0 (socket says 172.21.15.27:60778) 2024-04-16T05:56:14.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.034+0000 7f6761e00640 1 -- 172.21.15.27:0/262261871 learned_addr learned my addr 172.21.15.27:0/262261871 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:14.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.034+0000 7f6761400640 1 -- 172.21.15.27:0/262261871 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6764102040 msgr2=0x7f6764112b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:14.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.034+0000 7f6761400640 1 --2- 172.21.15.27:0/262261871 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6764102040 0x7f6764112b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:14.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.034+0000 7f6761400640 1 -- 172.21.15.27:0/262261871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6750005040 con 0x7f67641029e0 2024-04-16T05:56:14.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.035+0000 7f6761400640 1 --2- 172.21.15.27:0/262261871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67641029e0 0x7f67641130c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6750034040 tx=0x7f6750009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:14.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.036+0000 7f675a000640 1 -- 172.21.15.27:0/262261871 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67500094c0 con 0x7f67641029e0 2024-04-16T05:56:14.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.036+0000 7f6763200640 1 -- 172.21.15.27:0/262261871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676410cf90 con 0x7f67641029e0 2024-04-16T05:56:14.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.036+0000 7f675a000640 1 -- 172.21.15.27:0/262261871 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6750009620 con 0x7f67641029e0 2024-04-16T05:56:14.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.036+0000 7f675a000640 1 -- 172.21.15.27:0/262261871 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67500433b0 con 0x7f67641029e0 2024-04-16T05:56:14.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.036+0000 7f6763200640 1 -- 172.21.15.27:0/262261871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f676410d420 con 0x7f67641029e0 2024-04-16T05:56:14.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.038+0000 7f6763200640 1 -- 172.21.15.27:0/262261871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67640fd7e0 con 0x7f67641029e0 2024-04-16T05:56:14.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.043+0000 7f675a000640 1 -- 172.21.15.27:0/262261871 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f675002f070 con 0x7f67641029e0 2024-04-16T05:56:14.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.045+0000 7f675a000640 1 --2- 172.21.15.27:0/262261871 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f674806eb70 0x7f6748071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:14.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.045+0000 7f675a000640 1 -- 172.21.15.27:0/262261871 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f67500b0900 con 0x7f67641029e0 2024-04-16T05:56:14.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.045+0000 7f6761e00640 1 --2- 172.21.15.27:0/262261871 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f674806eb70 0x7f6748071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:14.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.047+0000 7f6761e00640 1 --2- 172.21.15.27:0/262261871 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f674806eb70 0x7f6748071020 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6754004e60 tx=0x7f6754009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:14.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.053+0000 7f675a000640 1 -- 172.21.15.27:0/262261871 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f67500794c0 con 0x7f67641029e0 2024-04-16T05:56:14.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:13 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/360542721' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 23, "format": "json"}]: dispatch 2024-04-16T05:56:14.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:13 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/3819209446' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 24, "format": "json"}]: dispatch 2024-04-16T05:56:14.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.300+0000 7f6763200640 1 -- 172.21.15.27:0/262261871 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 26, "format": "json"} v 0) v1 -- 0x7f67641091a0 con 0x7f67641029e0 2024-04-16T05:56:14.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.302+0000 7f675a000640 1 -- 172.21.15.27:0/262261871 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 26, "format": "json"}]=0 dumped fsmap epoch 26 v36) v1 ==== 107+0+4217 (secure 0 0 0) 0x7f675007cb80 con 0x7f67641029e0 2024-04-16T05:56:14.300 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:14.300 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34400,"name":"cephfs.smithi027.hkhpuq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/2831468256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2831468256},{"type":"v1","addr":"172.21.15.27:6827","nonce":2831468256}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":34432,"name":"cephfs.smithi027.ejoytk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6829/3219609959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3219609959},{"type":"v1","addr":"172.21.15.27:6829","nonce":3219609959}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T05:40:11.858650+0000","modified":"2024-04-16T05:51:15.659036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24273":{"gid":24273,"name":"cephfs.smithi122.rulewi","rank":0,"incarnation":26,"state":"up:replay","state_seq":2,"addr":"172.21.15.122:6827/1078307975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1078307975},{"type":"v1","addr":"172.21.15.122:6827","nonce":1078307975}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T05:56:14.300 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 26 2024-04-16T05:56:14.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.305+0000 7f6763200640 1 -- 172.21.15.27:0/262261871 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f674806eb70 msgr2=0x7f6748071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:14.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.305+0000 7f6763200640 1 --2- 172.21.15.27:0/262261871 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f674806eb70 0x7f6748071020 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6754004e60 tx=0x7f6754009040 comp rx=0 tx=0).stop 2024-04-16T05:56:14.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.305+0000 7f6763200640 1 -- 172.21.15.27:0/262261871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67641029e0 msgr2=0x7f67641130c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:14.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.305+0000 7f6763200640 1 --2- 172.21.15.27:0/262261871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67641029e0 0x7f67641130c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6750034040 tx=0x7f6750009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:14.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.305+0000 7f6763200640 1 -- 172.21.15.27:0/262261871 shutdown_connections 2024-04-16T05:56:14.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.305+0000 7f6763200640 1 --2- 172.21.15.27:0/262261871 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f674806eb70 0x7f6748071020 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:14.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.305+0000 7f6763200640 1 --2- 172.21.15.27:0/262261871 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67641029e0 0x7f67641130c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:14.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.305+0000 7f6763200640 1 --2- 172.21.15.27:0/262261871 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6764102040 0x7f6764112b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:14.303 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.305+0000 7f6763200640 1 -- 172.21.15.27:0/262261871 >> 172.21.15.27:0/262261871 conn(0x7f6764071010 msgr2=0x7f67640fa350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:14.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.305+0000 7f6763200640 1 -- 172.21.15.27:0/262261871 shutdown_connections 2024-04-16T05:56:14.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.305+0000 7f6763200640 1 -- 172.21.15.27:0/262261871 wait complete. 2024-04-16T05:56:14.323 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 27 2024-04-16T05:56:14.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.547+0000 7f1703e00640 1 -- 172.21.15.27:0/1291628485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16fc108860 msgr2=0x7f16fc108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:14.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.547+0000 7f1703e00640 1 --2- 172.21.15.27:0/1291628485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16fc108860 0x7f16fc108c30 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f16f00048d0 tx=0x7f16f0009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:14.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.547+0000 7f1703e00640 1 -- 172.21.15.27:0/1291628485 shutdown_connections 2024-04-16T05:56:14.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.547+0000 7f1703e00640 1 --2- 172.21.15.27:0/1291628485 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16fc102840 0x7f16fc102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:14.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.547+0000 7f1703e00640 1 --2- 172.21.15.27:0/1291628485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16fc108860 0x7f16fc108c30 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:14.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.547+0000 7f1703e00640 1 -- 172.21.15.27:0/1291628485 >> 172.21.15.27:0/1291628485 conn(0x7f16fc072050 msgr2=0x7f16fc074460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:14.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.547+0000 7f1703e00640 1 -- 172.21.15.27:0/1291628485 shutdown_connections 2024-04-16T05:56:14.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.547+0000 7f1703e00640 1 -- 172.21.15.27:0/1291628485 wait complete. 2024-04-16T05:56:14.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.548+0000 7f1703e00640 1 Processor -- start 2024-04-16T05:56:14.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.548+0000 7f1703e00640 1 -- start start 2024-04-16T05:56:14.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.549+0000 7f1703e00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16fc102840 0x7f16fc119280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:14.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.549+0000 7f1703e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16fc108860 0x7f16fc1197c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:14.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.549+0000 7f1703e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc119dc0 con 0x7f16fc108860 2024-04-16T05:56:14.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.549+0000 7f1703e00640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16fc119f30 con 0x7f16fc102840 2024-04-16T05:56:14.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.549+0000 7f1700a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16fc108860 0x7f16fc1197c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:14.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.549+0000 7f1700a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16fc108860 0x7f16fc1197c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58720/0 (socket says 172.21.15.27:58720) 2024-04-16T05:56:14.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.549+0000 7f1700a00640 1 -- 172.21.15.27:0/1689384659 learned_addr learned my addr 172.21.15.27:0/1689384659 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:14.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.549+0000 7f1701400640 1 --2- 172.21.15.27:0/1689384659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16fc102840 0x7f16fc119280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:14.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.550+0000 7f1700a00640 1 -- 172.21.15.27:0/1689384659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16fc102840 msgr2=0x7f16fc119280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:14.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.550+0000 7f1700a00640 1 --2- 172.21.15.27:0/1689384659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16fc102840 0x7f16fc119280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:14.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.550+0000 7f1700a00640 1 -- 172.21.15.27:0/1689384659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f0005040 con 0x7f16fc108860 2024-04-16T05:56:14.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.550+0000 7f1701400640 1 --2- 172.21.15.27:0/1689384659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16fc102840 0x7f16fc119280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-16T05:56:14.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.550+0000 7f1700a00640 1 --2- 172.21.15.27:0/1689384659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16fc108860 0x7f16fc1197c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f16e800c830 tx=0x7f16e8004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:14.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.550+0000 7f16f9600640 1 -- 172.21.15.27:0/1689384659 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16e800a850 con 0x7f16fc108860 2024-04-16T05:56:14.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.550+0000 7f16f9600640 1 -- 172.21.15.27:0/1689384659 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f16e800ae80 con 0x7f16fc108860 2024-04-16T05:56:14.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.550+0000 7f16f9600640 1 -- 172.21.15.27:0/1689384659 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f16e8012ea0 con 0x7f16fc108860 2024-04-16T05:56:14.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.550+0000 7f1703e00640 1 -- 172.21.15.27:0/1689384659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16fc1135c0 con 0x7f16fc108860 2024-04-16T05:56:14.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.551+0000 7f1703e00640 1 -- 172.21.15.27:0/1689384659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16fc113ae0 con 0x7f16fc108860 2024-04-16T05:56:14.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.552+0000 7f1703e00640 1 -- 172.21.15.27:0/1689384659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16fc103f80 con 0x7f16fc108860 2024-04-16T05:56:14.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.553+0000 7f16f9600640 1 -- 172.21.15.27:0/1689384659 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f16e8012400 con 0x7f16fc108860 2024-04-16T05:56:14.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.554+0000 7f16f9600640 1 --2- 172.21.15.27:0/1689384659 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f16e406eb70 0x7f16e4071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:14.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.554+0000 7f16f9600640 1 -- 172.21.15.27:0/1689384659 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f16e808ecc0 con 0x7f16fc108860 2024-04-16T05:56:14.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.555+0000 7f1701400640 1 --2- 172.21.15.27:0/1689384659 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f16e406eb70 0x7f16e4071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:14.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.555+0000 7f1701400640 1 --2- 172.21.15.27:0/1689384659 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f16e406eb70 0x7f16e4071020 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f16f00048a0 tx=0x7f16f0009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:14.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.559+0000 7f16f9600640 1 -- 172.21.15.27:0/1689384659 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f16e8094020 con 0x7f16fc108860 2024-04-16T05:56:14.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.805+0000 7f1703e00640 1 -- 172.21.15.27:0/1689384659 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 27, "format": "json"} v 0) v1 -- 0x7f16fc10f9d0 con 0x7f16fc108860 2024-04-16T05:56:14.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.807+0000 7f16f9600640 1 -- 172.21.15.27:0/1689384659 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 27, "format": "json"}]=0 dumped fsmap epoch 27 v36) v1 ==== 107+0+4222 (secure 0 0 0) 0x7f16e80578d0 con 0x7f16fc108860 2024-04-16T05:56:14.804 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:14.804 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34400,"name":"cephfs.smithi027.hkhpuq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/2831468256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2831468256},{"type":"v1","addr":"172.21.15.27:6827","nonce":2831468256}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":34432,"name":"cephfs.smithi027.ejoytk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6829/3219609959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3219609959},{"type":"v1","addr":"172.21.15.27:6829","nonce":3219609959}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T05:40:11.858650+0000","modified":"2024-04-16T05:51:15.681873+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24273":{"gid":24273,"name":"cephfs.smithi122.rulewi","rank":0,"incarnation":26,"state":"up:reconnect","state_seq":163,"addr":"172.21.15.122:6827/1078307975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1078307975},{"type":"v1","addr":"172.21.15.122:6827","nonce":1078307975}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T05:56:14.805 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 27 2024-04-16T05:56:14.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.809+0000 7f1703e00640 1 -- 172.21.15.27:0/1689384659 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f16e406eb70 msgr2=0x7f16e4071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:14.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.809+0000 7f1703e00640 1 --2- 172.21.15.27:0/1689384659 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f16e406eb70 0x7f16e4071020 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f16f00048a0 tx=0x7f16f0009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:14.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.809+0000 7f1703e00640 1 -- 172.21.15.27:0/1689384659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16fc108860 msgr2=0x7f16fc1197c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:14.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.809+0000 7f1703e00640 1 --2- 172.21.15.27:0/1689384659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16fc108860 0x7f16fc1197c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f16e800c830 tx=0x7f16e8004430 comp rx=0 tx=0).stop 2024-04-16T05:56:14.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.810+0000 7f1703e00640 1 -- 172.21.15.27:0/1689384659 shutdown_connections 2024-04-16T05:56:14.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.810+0000 7f1703e00640 1 --2- 172.21.15.27:0/1689384659 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f16e406eb70 0x7f16e4071020 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:14.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.810+0000 7f1703e00640 1 --2- 172.21.15.27:0/1689384659 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f16fc108860 0x7f16fc1197c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:14.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.810+0000 7f1703e00640 1 --2- 172.21.15.27:0/1689384659 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f16fc102840 0x7f16fc119280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:14.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.810+0000 7f1703e00640 1 -- 172.21.15.27:0/1689384659 >> 172.21.15.27:0/1689384659 conn(0x7f16fc072050 msgr2=0x7f16fc073fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:14.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.810+0000 7f1703e00640 1 -- 172.21.15.27:0/1689384659 shutdown_connections 2024-04-16T05:56:14.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:14.810+0000 7f1703e00640 1 -- 172.21.15.27:0/1689384659 wait complete. 2024-04-16T05:56:14.826 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 28 2024-04-16T05:56:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:14 smithi122 ceph-mon[80930]: pgmap v364: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:14 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1249866857' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-16T05:56:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:56:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:14 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/262261871' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-16T05:56:15.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.042+0000 7fc0f2200640 1 -- 172.21.15.27:0/807375566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0ec1075a0 msgr2=0x7fc0ec107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:15.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.042+0000 7fc0f2200640 1 --2- 172.21.15.27:0/807375566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0ec1075a0 0x7fc0ec107970 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc0e00048d0 tx=0x7fc0e0009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:15.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.042+0000 7fc0f2200640 1 -- 172.21.15.27:0/807375566 shutdown_connections 2024-04-16T05:56:15.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.042+0000 7fc0f2200640 1 --2- 172.21.15.27:0/807375566 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0ec102670 0x7fc0ec102ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:15.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.042+0000 7fc0f2200640 1 --2- 172.21.15.27:0/807375566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0ec1075a0 0x7fc0ec107970 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:15.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.042+0000 7fc0f2200640 1 -- 172.21.15.27:0/807375566 >> 172.21.15.27:0/807375566 conn(0x7fc0ec071e60 msgr2=0x7fc0ec074270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:15.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.042+0000 7fc0f2200640 1 -- 172.21.15.27:0/807375566 shutdown_connections 2024-04-16T05:56:15.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.042+0000 7fc0f2200640 1 -- 172.21.15.27:0/807375566 wait complete. 2024-04-16T05:56:15.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.043+0000 7fc0f2200640 1 Processor -- start 2024-04-16T05:56:15.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.043+0000 7fc0f2200640 1 -- start start 2024-04-16T05:56:15.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.043+0000 7fc0f2200640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0ec102670 0x7fc0ec1191d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:15.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.043+0000 7fc0f2200640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0ec1075a0 0x7fc0ec119710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:15.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.043+0000 7fc0f2200640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ec119da0 con 0x7fc0ec102670 2024-04-16T05:56:15.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.043+0000 7fc0f2200640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ec113360 con 0x7fc0ec1075a0 2024-04-16T05:56:15.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.043+0000 7fc0f0e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0ec102670 0x7fc0ec1191d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:15.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.043+0000 7fc0f0e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0ec102670 0x7fc0ec1191d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58748/0 (socket says 172.21.15.27:58748) 2024-04-16T05:56:15.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.043+0000 7fc0f0e00640 1 -- 172.21.15.27:0/2413286493 learned_addr learned my addr 172.21.15.27:0/2413286493 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:15.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.044+0000 7fc0f0e00640 1 -- 172.21.15.27:0/2413286493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0ec1075a0 msgr2=0x7fc0ec119710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:15.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.044+0000 7fc0ebe00640 1 --2- 172.21.15.27:0/2413286493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0ec1075a0 0x7fc0ec119710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:15.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.044+0000 7fc0f0e00640 1 --2- 172.21.15.27:0/2413286493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0ec1075a0 0x7fc0ec119710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:15.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.044+0000 7fc0f0e00640 1 -- 172.21.15.27:0/2413286493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e0005040 con 0x7fc0ec102670 2024-04-16T05:56:15.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.044+0000 7fc0ebe00640 1 --2- 172.21.15.27:0/2413286493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0ec1075a0 0x7fc0ec119710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:56:15.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.044+0000 7fc0f0e00640 1 --2- 172.21.15.27:0/2413286493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0ec102670 0x7fc0ec1191d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc0e002be70 tx=0x7fc0e0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:15.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.045+0000 7fc0e8c00640 1 -- 172.21.15.27:0/2413286493 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0e00094c0 con 0x7fc0ec102670 2024-04-16T05:56:15.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.045+0000 7fc0e8c00640 1 -- 172.21.15.27:0/2413286493 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc0e0009620 con 0x7fc0ec102670 2024-04-16T05:56:15.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.045+0000 7fc0f2200640 1 -- 172.21.15.27:0/2413286493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0ec1135e0 con 0x7fc0ec102670 2024-04-16T05:56:15.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.045+0000 7fc0e8c00640 1 -- 172.21.15.27:0/2413286493 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc0e00435f0 con 0x7fc0ec102670 2024-04-16T05:56:15.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.045+0000 7fc0f2200640 1 -- 172.21.15.27:0/2413286493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0ec113aa0 con 0x7fc0ec102670 2024-04-16T05:56:15.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.046+0000 7fc0f2200640 1 -- 172.21.15.27:0/2413286493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0ec103d60 con 0x7fc0ec102670 2024-04-16T05:56:15.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.047+0000 7fc0e8c00640 1 -- 172.21.15.27:0/2413286493 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc0e002f070 con 0x7fc0ec102670 2024-04-16T05:56:15.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.048+0000 7fc0e8c00640 1 --2- 172.21.15.27:0/2413286493 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc0d406eb70 0x7fc0d4071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:15.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.048+0000 7fc0e8c00640 1 -- 172.21.15.27:0/2413286493 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7fc0e00b0830 con 0x7fc0ec102670 2024-04-16T05:56:15.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.048+0000 7fc0ebe00640 1 --2- 172.21.15.27:0/2413286493 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc0d406eb70 0x7fc0d4071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:15.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.049+0000 7fc0ebe00640 1 --2- 172.21.15.27:0/2413286493 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc0d406eb70 0x7fc0d4071020 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8006fd0 tx=0x7fc0d8008040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:15.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.053+0000 7fc0e8c00640 1 -- 172.21.15.27:0/2413286493 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc0e00793e0 con 0x7fc0ec102670 2024-04-16T05:56:15.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:14 smithi027 ceph-mon[93635]: pgmap v364: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:15.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:14 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1249866857' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 25, "format": "json"}]: dispatch 2024-04-16T05:56:15.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:56:15.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:14 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/262261871' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 26, "format": "json"}]: dispatch 2024-04-16T05:56:15.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.301+0000 7fc0f2200640 1 -- 172.21.15.27:0/2413286493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 28, "format": "json"} v 0) v1 -- 0x7fc0ec1148e0 con 0x7fc0ec102670 2024-04-16T05:56:15.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.303+0000 7fc0e8c00640 1 -- 172.21.15.27:0/2413286493 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 28, "format": "json"}]=0 dumped fsmap epoch 28 v36) v1 ==== 107+0+4219 (secure 0 0 0) 0x7fc0e007caa0 con 0x7fc0ec102670 2024-04-16T05:56:15.301 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:15.301 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34400,"name":"cephfs.smithi027.hkhpuq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/2831468256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2831468256},{"type":"v1","addr":"172.21.15.27:6827","nonce":2831468256}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":34432,"name":"cephfs.smithi027.ejoytk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6829/3219609959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3219609959},{"type":"v1","addr":"172.21.15.27:6829","nonce":3219609959}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T05:40:11.858650+0000","modified":"2024-04-16T05:51:16.673472+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24273":{"gid":24273,"name":"cephfs.smithi122.rulewi","rank":0,"incarnation":26,"state":"up:rejoin","state_seq":164,"addr":"172.21.15.122:6827/1078307975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1078307975},{"type":"v1","addr":"172.21.15.122:6827","nonce":1078307975}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T05:56:15.301 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 28 2024-04-16T05:56:15.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.305+0000 7fc0f2200640 1 -- 172.21.15.27:0/2413286493 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc0d406eb70 msgr2=0x7fc0d4071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:15.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.305+0000 7fc0f2200640 1 --2- 172.21.15.27:0/2413286493 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc0d406eb70 0x7fc0d4071020 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8006fd0 tx=0x7fc0d8008040 comp rx=0 tx=0).stop 2024-04-16T05:56:15.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.305+0000 7fc0f2200640 1 -- 172.21.15.27:0/2413286493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0ec102670 msgr2=0x7fc0ec1191d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:15.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.305+0000 7fc0f2200640 1 --2- 172.21.15.27:0/2413286493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0ec102670 0x7fc0ec1191d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc0e002be70 tx=0x7fc0e0009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:15.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.306+0000 7fc0f2200640 1 -- 172.21.15.27:0/2413286493 shutdown_connections 2024-04-16T05:56:15.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.306+0000 7fc0f2200640 1 --2- 172.21.15.27:0/2413286493 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc0d406eb70 0x7fc0d4071020 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:15.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.306+0000 7fc0f2200640 1 --2- 172.21.15.27:0/2413286493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0ec1075a0 0x7fc0ec119710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:15.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.306+0000 7fc0f2200640 1 --2- 172.21.15.27:0/2413286493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc0ec102670 0x7fc0ec1191d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:15.304 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.306+0000 7fc0f2200640 1 -- 172.21.15.27:0/2413286493 >> 172.21.15.27:0/2413286493 conn(0x7fc0ec071e60 msgr2=0x7fc0ec104d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:15.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.306+0000 7fc0f2200640 1 -- 172.21.15.27:0/2413286493 shutdown_connections 2024-04-16T05:56:15.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.306+0000 7fc0f2200640 1 -- 172.21.15.27:0/2413286493 wait complete. 2024-04-16T05:56:15.324 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 29 2024-04-16T05:56:15.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.559+0000 7f181a800640 1 -- 172.21.15.27:0/1360390299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1814101fe0 msgr2=0x7f18141023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:15.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.559+0000 7f181a800640 1 --2- 172.21.15.27:0/1360390299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1814101fe0 0x7f18141023b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f18080048d0 tx=0x7f1808009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:15.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.559+0000 7f181a800640 1 -- 172.21.15.27:0/1360390299 shutdown_connections 2024-04-16T05:56:15.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.559+0000 7f181a800640 1 --2- 172.21.15.27:0/1360390299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1814102980 0x7f18140fc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:15.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.559+0000 7f181a800640 1 --2- 172.21.15.27:0/1360390299 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1814101fe0 0x7f18141023b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:15.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.559+0000 7f181a800640 1 -- 172.21.15.27:0/1360390299 >> 172.21.15.27:0/1360390299 conn(0x7f18140f7b90 msgr2=0x7f18140f9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:15.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.559+0000 7f181a800640 1 -- 172.21.15.27:0/1360390299 shutdown_connections 2024-04-16T05:56:15.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.559+0000 7f181a800640 1 -- 172.21.15.27:0/1360390299 wait complete. 2024-04-16T05:56:15.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.560+0000 7f181a800640 1 Processor -- start 2024-04-16T05:56:15.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.560+0000 7f181a800640 1 -- start start 2024-04-16T05:56:15.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.560+0000 7f181a800640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1814101fe0 0x7f1814114cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:15.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.560+0000 7f181a800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1814102980 0x7f1814115210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:15.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.560+0000 7f181a800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1814115810 con 0x7f1814102980 2024-04-16T05:56:15.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.560+0000 7f181a800640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181410ee60 con 0x7f1814101fe0 2024-04-16T05:56:15.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.561+0000 7f1818a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1814102980 0x7f1814115210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:15.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.561+0000 7f1818a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1814102980 0x7f1814115210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58756/0 (socket says 172.21.15.27:58756) 2024-04-16T05:56:15.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.561+0000 7f1818a00640 1 -- 172.21.15.27:0/2977367302 learned_addr learned my addr 172.21.15.27:0/2977367302 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:15.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.561+0000 7f1819400640 1 --2- 172.21.15.27:0/2977367302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1814101fe0 0x7f1814114cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:15.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.561+0000 7f1818a00640 1 -- 172.21.15.27:0/2977367302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1814101fe0 msgr2=0x7f1814114cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:15.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.561+0000 7f1818a00640 1 --2- 172.21.15.27:0/2977367302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1814101fe0 0x7f1814114cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:15.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.561+0000 7f1818a00640 1 -- 172.21.15.27:0/2977367302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1808005040 con 0x7f1814102980 2024-04-16T05:56:15.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.561+0000 7f1819400640 1 --2- 172.21.15.27:0/2977367302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1814101fe0 0x7f1814114cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:56:15.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.561+0000 7f1818a00640 1 --2- 172.21.15.27:0/2977367302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1814102980 0x7f1814115210 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1800002a50 tx=0x7f1800004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:15.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.562+0000 7f1811600640 1 -- 172.21.15.27:0/2977367302 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1800007c30 con 0x7f1814102980 2024-04-16T05:56:15.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.562+0000 7f1811600640 1 -- 172.21.15.27:0/2977367302 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f180000f040 con 0x7f1814102980 2024-04-16T05:56:15.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.562+0000 7f181a800640 1 -- 172.21.15.27:0/2977367302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181410f140 con 0x7f1814102980 2024-04-16T05:56:15.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.562+0000 7f1811600640 1 -- 172.21.15.27:0/2977367302 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f180000db50 con 0x7f1814102980 2024-04-16T05:56:15.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.562+0000 7f181a800640 1 -- 172.21.15.27:0/2977367302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f181410f660 con 0x7f1814102980 2024-04-16T05:56:15.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.563+0000 7f181a800640 1 -- 172.21.15.27:0/2977367302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18140fd750 con 0x7f1814102980 2024-04-16T05:56:15.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.564+0000 7f1811600640 1 -- 172.21.15.27:0/2977367302 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f180000dcf0 con 0x7f1814102980 2024-04-16T05:56:15.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.565+0000 7f1811600640 1 --2- 172.21.15.27:0/2977367302 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f17fc06eb70 0x7f17fc071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:15.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.565+0000 7f1811600640 1 -- 172.21.15.27:0/2977367302 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f180008e2e0 con 0x7f1814102980 2024-04-16T05:56:15.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.566+0000 7f1819400640 1 --2- 172.21.15.27:0/2977367302 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f17fc06eb70 0x7f17fc071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:15.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.567+0000 7f1819400640 1 --2- 172.21.15.27:0/2977367302 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f17fc06eb70 0x7f17fc071020 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1808000c00 tx=0x7f1808009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:15.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.570+0000 7f1811600640 1 -- 172.21.15.27:0/2977367302 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f1800056f10 con 0x7f1814102980 2024-04-16T05:56:15.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.817+0000 7f181a800640 1 -- 172.21.15.27:0/2977367302 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 29, "format": "json"} v 0) v1 -- 0x7f18141091a0 con 0x7f1814102980 2024-04-16T05:56:15.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.818+0000 7f1811600640 1 -- 172.21.15.27:0/2977367302 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 29, "format": "json"}]=0 dumped fsmap epoch 29 v36) v1 ==== 107+0+4228 (secure 0 0 0) 0x7f180005a5d0 con 0x7f1814102980 2024-04-16T05:56:15.816 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:15.817 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34400,"name":"cephfs.smithi027.hkhpuq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/2831468256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2831468256},{"type":"v1","addr":"172.21.15.27:6827","nonce":2831468256}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":34432,"name":"cephfs.smithi027.ejoytk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6829/3219609959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3219609959},{"type":"v1","addr":"172.21.15.27:6829","nonce":3219609959}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T05:40:11.858650+0000","modified":"2024-04-16T05:51:18.679503+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24273":{"gid":24273,"name":"cephfs.smithi122.rulewi","rank":0,"incarnation":26,"state":"up:active","state_seq":165,"addr":"172.21.15.122:6827/1078307975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1078307975},{"type":"v1","addr":"172.21.15.122:6827","nonce":1078307975}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24273,"qdb_cluster":[24273]},"id":1}]} 2024-04-16T05:56:15.817 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 29 2024-04-16T05:56:15.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.821+0000 7f181a800640 1 -- 172.21.15.27:0/2977367302 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f17fc06eb70 msgr2=0x7f17fc071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:15.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.821+0000 7f181a800640 1 --2- 172.21.15.27:0/2977367302 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f17fc06eb70 0x7f17fc071020 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1808000c00 tx=0x7f1808009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:15.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.821+0000 7f181a800640 1 -- 172.21.15.27:0/2977367302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1814102980 msgr2=0x7f1814115210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:15.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.821+0000 7f181a800640 1 --2- 172.21.15.27:0/2977367302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1814102980 0x7f1814115210 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1800002a50 tx=0x7f1800004600 comp rx=0 tx=0).stop 2024-04-16T05:56:15.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.821+0000 7f181a800640 1 -- 172.21.15.27:0/2977367302 shutdown_connections 2024-04-16T05:56:15.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.821+0000 7f181a800640 1 --2- 172.21.15.27:0/2977367302 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f17fc06eb70 0x7f17fc071020 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:15.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.821+0000 7f181a800640 1 --2- 172.21.15.27:0/2977367302 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1814102980 0x7f1814115210 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:15.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.821+0000 7f181a800640 1 --2- 172.21.15.27:0/2977367302 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1814101fe0 0x7f1814114cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:15.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.821+0000 7f181a800640 1 -- 172.21.15.27:0/2977367302 >> 172.21.15.27:0/2977367302 conn(0x7f18140f7b90 msgr2=0x7f1814071160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:15.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.822+0000 7f181a800640 1 -- 172.21.15.27:0/2977367302 shutdown_connections 2024-04-16T05:56:15.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:15.822+0000 7f181a800640 1 -- 172.21.15.27:0/2977367302 wait complete. 2024-04-16T05:56:15.837 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 30 2024-04-16T05:56:16.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:15 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1689384659' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-16T05:56:16.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:15 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/2413286493' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-16T05:56:16.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:15 smithi122 ceph-mon[80930]: pgmap v365: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:16.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:15 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1689384659' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 27, "format": "json"}]: dispatch 2024-04-16T05:56:16.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:15 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/2413286493' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 28, "format": "json"}]: dispatch 2024-04-16T05:56:16.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:15 smithi027 ceph-mon[93635]: pgmap v365: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:16.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.069+0000 7f6c52c00640 1 -- 172.21.15.27:0/2976871833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c4c0fb840 msgr2=0x7f6c4c1085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:16.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.069+0000 7f6c52c00640 1 --2- 172.21.15.27:0/2976871833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c4c0fb840 0x7f6c4c1085a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6c380048d0 tx=0x7f6c38009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:16.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.069+0000 7f6c52c00640 1 -- 172.21.15.27:0/2976871833 shutdown_connections 2024-04-16T05:56:16.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.069+0000 7f6c52c00640 1 --2- 172.21.15.27:0/2976871833 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c4c0fb840 0x7f6c4c1085a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:16.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.069+0000 7f6c52c00640 1 --2- 172.21.15.27:0/2976871833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c4c0faf30 0x7f6c4c0fb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:16.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.069+0000 7f6c52c00640 1 -- 172.21.15.27:0/2976871833 >> 172.21.15.27:0/2976871833 conn(0x7f6c4c0f6c20 msgr2=0x7f6c4c0f9030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:16.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.069+0000 7f6c52c00640 1 -- 172.21.15.27:0/2976871833 shutdown_connections 2024-04-16T05:56:16.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.069+0000 7f6c52c00640 1 -- 172.21.15.27:0/2976871833 wait complete. 2024-04-16T05:56:16.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.070+0000 7f6c52c00640 1 Processor -- start 2024-04-16T05:56:16.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.070+0000 7f6c52c00640 1 -- start start 2024-04-16T05:56:16.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.070+0000 7f6c52c00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c4c0faf30 0x7f6c4c114e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:16.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.070+0000 7f6c52c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c4c0fb840 0x7f6c4c1153a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:16.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.070+0000 7f6c52c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c1159a0 con 0x7f6c4c0fb840 2024-04-16T05:56:16.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.070+0000 7f6c52c00640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c10eff0 con 0x7f6c4c0faf30 2024-04-16T05:56:16.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.071+0000 7f6c4b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c4c0fb840 0x7f6c4c1153a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:16.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.071+0000 7f6c4b400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c4c0fb840 0x7f6c4c1153a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58762/0 (socket says 172.21.15.27:58762) 2024-04-16T05:56:16.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.071+0000 7f6c4b400640 1 -- 172.21.15.27:0/2615735796 learned_addr learned my addr 172.21.15.27:0/2615735796 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:16.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.071+0000 7f6c4b400640 1 -- 172.21.15.27:0/2615735796 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c4c0faf30 msgr2=0x7f6c4c114e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:16.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.071+0000 7f6c4be00640 1 --2- 172.21.15.27:0/2615735796 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c4c0faf30 0x7f6c4c114e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:16.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.071+0000 7f6c4b400640 1 --2- 172.21.15.27:0/2615735796 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c4c0faf30 0x7f6c4c114e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:16.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.071+0000 7f6c4b400640 1 -- 172.21.15.27:0/2615735796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c38005040 con 0x7f6c4c0fb840 2024-04-16T05:56:16.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.072+0000 7f6c4b400640 1 --2- 172.21.15.27:0/2615735796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c4c0fb840 0x7f6c4c1153a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6c38003080 tx=0x7f6c38009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:16.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.072+0000 7f6c43e00640 1 -- 172.21.15.27:0/2615735796 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c3803b7f0 con 0x7f6c4c0fb840 2024-04-16T05:56:16.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.072+0000 7f6c43e00640 1 -- 172.21.15.27:0/2615735796 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c38031780 con 0x7f6c4c0fb840 2024-04-16T05:56:16.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.072+0000 7f6c52c00640 1 -- 172.21.15.27:0/2615735796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c4c10f270 con 0x7f6c4c0fb840 2024-04-16T05:56:16.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.072+0000 7f6c43e00640 1 -- 172.21.15.27:0/2615735796 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c380439a0 con 0x7f6c4c0fb840 2024-04-16T05:56:16.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.072+0000 7f6c52c00640 1 -- 172.21.15.27:0/2615735796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c4c10f810 con 0x7f6c4c0fb840 2024-04-16T05:56:16.071 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.073+0000 7f6c52c00640 1 -- 172.21.15.27:0/2615735796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c4c0660f0 con 0x7f6c4c0fb840 2024-04-16T05:56:16.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.074+0000 7f6c43e00640 1 -- 172.21.15.27:0/2615735796 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6c38043b20 con 0x7f6c4c0fb840 2024-04-16T05:56:16.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.075+0000 7f6c43e00640 1 --2- 172.21.15.27:0/2615735796 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6c3006eb70 0x7f6c30071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:16.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.075+0000 7f6c43e00640 1 -- 172.21.15.27:0/2615735796 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f6c380b1280 con 0x7f6c4c0fb840 2024-04-16T05:56:16.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.076+0000 7f6c4be00640 1 --2- 172.21.15.27:0/2615735796 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6c3006eb70 0x7f6c30071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:16.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.076+0000 7f6c4be00640 1 --2- 172.21.15.27:0/2615735796 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6c3006eb70 0x7f6c30071020 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c007fd0 tx=0x7f6c3c009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:16.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.082+0000 7f6c43e00640 1 -- 172.21.15.27:0/2615735796 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6c38079e10 con 0x7f6c4c0fb840 2024-04-16T05:56:16.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.331+0000 7f6c52c00640 1 -- 172.21.15.27:0/2615735796 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 30, "format": "json"} v 0) v1 -- 0x7f6c4c10fa80 con 0x7f6c4c0fb840 2024-04-16T05:56:16.330 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.333+0000 7f6c43e00640 1 -- 172.21.15.27:0/2615735796 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 30, "format": "json"}]=0 dumped fsmap epoch 30 v36) v1 ==== 107+0+5072 (secure 0 0 0) 0x7f6c3807d4d0 con 0x7f6c4c0fb840 2024-04-16T05:56:16.331 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:16.331 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34400,"name":"cephfs.smithi027.hkhpuq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/2831468256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2831468256},{"type":"v1","addr":"172.21.15.27:6827","nonce":2831468256}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":34432,"name":"cephfs.smithi027.ejoytk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6829/3219609959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3219609959},{"type":"v1","addr":"172.21.15.27:6829","nonce":3219609959}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":44337,"name":"cephfs.smithi122.lojjdp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6825/97103507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":97103507},{"type":"v1","addr":"172.21.15.122:6825","nonce":97103507}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T05:40:11.858650+0000","modified":"2024-04-16T05:51:18.679503+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":87,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24273":{"gid":24273,"name":"cephfs.smithi122.rulewi","rank":0,"incarnation":26,"state":"up:active","state_seq":165,"addr":"172.21.15.122:6827/1078307975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1078307975},{"type":"v1","addr":"172.21.15.122:6827","nonce":1078307975}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24273,"qdb_cluster":[24273]},"id":1}]} 2024-04-16T05:56:16.331 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 30 2024-04-16T05:56:16.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.335+0000 7f6c52c00640 1 -- 172.21.15.27:0/2615735796 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6c3006eb70 msgr2=0x7f6c30071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:16.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.335+0000 7f6c52c00640 1 --2- 172.21.15.27:0/2615735796 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6c3006eb70 0x7f6c30071020 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c007fd0 tx=0x7f6c3c009040 comp rx=0 tx=0).stop 2024-04-16T05:56:16.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.335+0000 7f6c52c00640 1 -- 172.21.15.27:0/2615735796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c4c0fb840 msgr2=0x7f6c4c1153a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:16.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.335+0000 7f6c52c00640 1 --2- 172.21.15.27:0/2615735796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c4c0fb840 0x7f6c4c1153a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6c38003080 tx=0x7f6c38009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:16.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.335+0000 7f6c52c00640 1 -- 172.21.15.27:0/2615735796 shutdown_connections 2024-04-16T05:56:16.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.335+0000 7f6c52c00640 1 --2- 172.21.15.27:0/2615735796 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6c3006eb70 0x7f6c30071020 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:16.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.335+0000 7f6c52c00640 1 --2- 172.21.15.27:0/2615735796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c4c0fb840 0x7f6c4c1153a0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:16.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.335+0000 7f6c52c00640 1 --2- 172.21.15.27:0/2615735796 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c4c0faf30 0x7f6c4c114e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:16.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.335+0000 7f6c52c00640 1 -- 172.21.15.27:0/2615735796 >> 172.21.15.27:0/2615735796 conn(0x7f6c4c0f6c20 msgr2=0x7f6c4c0f7050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:16.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.336+0000 7f6c52c00640 1 -- 172.21.15.27:0/2615735796 shutdown_connections 2024-04-16T05:56:16.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.336+0000 7f6c52c00640 1 -- 172.21.15.27:0/2615735796 wait complete. 2024-04-16T05:56:16.351 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 31 2024-04-16T05:56:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.572+0000 7fbe6c000640 1 -- 172.21.15.27:0/1887647333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe64071a00 msgr2=0x7fbe64071dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.572+0000 7fbe6c000640 1 --2- 172.21.15.27:0/1887647333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe64071a00 0x7fbe64071dd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbe54009c50 tx=0x7fbe54004d10 comp rx=0 tx=0).stop 2024-04-16T05:56:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.572+0000 7fbe6c000640 1 -- 172.21.15.27:0/1887647333 shutdown_connections 2024-04-16T05:56:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.572+0000 7fbe6c000640 1 --2- 172.21.15.27:0/1887647333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe64072310 0x7fbe6410cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.572+0000 7fbe6c000640 1 --2- 172.21.15.27:0/1887647333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe64071a00 0x7fbe64071dd0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.572+0000 7fbe6c000640 1 -- 172.21.15.27:0/1887647333 >> 172.21.15.27:0/1887647333 conn(0x7fbe640fa1b0 msgr2=0x7fbe640fc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:16.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.572+0000 7fbe6c000640 1 -- 172.21.15.27:0/1887647333 shutdown_connections 2024-04-16T05:56:16.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.572+0000 7fbe6c000640 1 -- 172.21.15.27:0/1887647333 wait complete. 2024-04-16T05:56:16.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.573+0000 7fbe6c000640 1 Processor -- start 2024-04-16T05:56:16.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.573+0000 7fbe6c000640 1 -- start start 2024-04-16T05:56:16.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.573+0000 7fbe6c000640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe64071a00 0x7fbe641194d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:16.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.573+0000 7fbe6c000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe64072310 0x7fbe64119a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:16.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.573+0000 7fbe6c000640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6411a060 con 0x7fbe64072310 2024-04-16T05:56:16.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.573+0000 7fbe6c000640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe64113680 con 0x7fbe64071a00 2024-04-16T05:56:16.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.574+0000 7fbe68c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe64072310 0x7fbe64119a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:16.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.574+0000 7fbe68c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe64072310 0x7fbe64119a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58786/0 (socket says 172.21.15.27:58786) 2024-04-16T05:56:16.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.574+0000 7fbe68c00640 1 -- 172.21.15.27:0/1949872951 learned_addr learned my addr 172.21.15.27:0/1949872951 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:16.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.574+0000 7fbe69600640 1 --2- 172.21.15.27:0/1949872951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe64071a00 0x7fbe641194d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:16.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.574+0000 7fbe68c00640 1 -- 172.21.15.27:0/1949872951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe64071a00 msgr2=0x7fbe641194d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:16.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.574+0000 7fbe68c00640 1 --2- 172.21.15.27:0/1949872951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe64071a00 0x7fbe641194d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:16.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.574+0000 7fbe68c00640 1 -- 172.21.15.27:0/1949872951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe540098b0 con 0x7fbe64072310 2024-04-16T05:56:16.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.574+0000 7fbe69600640 1 --2- 172.21.15.27:0/1949872951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe64071a00 0x7fbe641194d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:56:16.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.575+0000 7fbe68c00640 1 --2- 172.21.15.27:0/1949872951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe64072310 0x7fbe64119a10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fbe50002a50 tx=0x7fbe50004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:16.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.575+0000 7fbe61600640 1 -- 172.21.15.27:0/1949872951 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe50007c30 con 0x7fbe64072310 2024-04-16T05:56:16.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.575+0000 7fbe61600640 1 -- 172.21.15.27:0/1949872951 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe5000f040 con 0x7fbe64072310 2024-04-16T05:56:16.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.575+0000 7fbe61600640 1 -- 172.21.15.27:0/1949872951 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe5000db50 con 0x7fbe64072310 2024-04-16T05:56:16.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.575+0000 7fbe6c000640 1 -- 172.21.15.27:0/1949872951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe64113960 con 0x7fbe64072310 2024-04-16T05:56:16.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.575+0000 7fbe6c000640 1 -- 172.21.15.27:0/1949872951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe64113f00 con 0x7fbe64072310 2024-04-16T05:56:16.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.576+0000 7fbe6c000640 1 -- 172.21.15.27:0/1949872951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe64073140 con 0x7fbe64072310 2024-04-16T05:56:16.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.577+0000 7fbe61600640 1 -- 172.21.15.27:0/1949872951 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fbe5000dcf0 con 0x7fbe64072310 2024-04-16T05:56:16.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.578+0000 7fbe61600640 1 --2- 172.21.15.27:0/1949872951 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fbe4806eb70 0x7fbe48071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:16.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.578+0000 7fbe61600640 1 -- 172.21.15.27:0/1949872951 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7fbe5008e380 con 0x7fbe64072310 2024-04-16T05:56:16.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.579+0000 7fbe69600640 1 --2- 172.21.15.27:0/1949872951 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fbe4806eb70 0x7fbe48071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.579+0000 7fbe69600640 1 --2- 172.21.15.27:0/1949872951 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fbe4806eb70 0x7fbe48071020 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbe54004a30 tx=0x7fbe5402e040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.583+0000 7fbe61600640 1 -- 172.21.15.27:0/1949872951 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fbe50056f10 con 0x7fbe64072310 2024-04-16T05:56:16.828 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.830+0000 7fbe6c000640 1 -- 172.21.15.27:0/1949872951 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 31, "format": "json"} v 0) v1 -- 0x7fbe6410fb50 con 0x7fbe64072310 2024-04-16T05:56:16.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.832+0000 7fbe61600640 1 -- 172.21.15.27:0/1949872951 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 31, "format": "json"}]=0 dumped fsmap epoch 31 v36) v1 ==== 107+0+4268 (secure 0 0 0) 0x7fbe5005a5d0 con 0x7fbe64072310 2024-04-16T05:56:16.830 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:16.830 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34400,"name":"cephfs.smithi027.hkhpuq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6827/2831468256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2831468256},{"type":"v1","addr":"172.21.15.27:6827","nonce":2831468256}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":34432,"name":"cephfs.smithi027.ejoytk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6829/3219609959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3219609959},{"type":"v1","addr":"172.21.15.27:6829","nonce":3219609959}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":44337,"name":"cephfs.smithi122.lojjdp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6825/97103507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":97103507},{"type":"v1","addr":"172.21.15.122:6825","nonce":97103507}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T05:40:11.858650+0000","modified":"2024-04-16T05:51:44.011079+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T05:56:16.830 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 31 2024-04-16T05:56:16.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.834+0000 7fbe6c000640 1 -- 172.21.15.27:0/1949872951 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fbe4806eb70 msgr2=0x7fbe48071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:16.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.834+0000 7fbe6c000640 1 --2- 172.21.15.27:0/1949872951 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fbe4806eb70 0x7fbe48071020 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbe54004a30 tx=0x7fbe5402e040 comp rx=0 tx=0).stop 2024-04-16T05:56:16.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.834+0000 7fbe6c000640 1 -- 172.21.15.27:0/1949872951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe64072310 msgr2=0x7fbe64119a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:16.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.834+0000 7fbe6c000640 1 --2- 172.21.15.27:0/1949872951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe64072310 0x7fbe64119a10 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fbe50002a50 tx=0x7fbe50004600 comp rx=0 tx=0).stop 2024-04-16T05:56:16.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.835+0000 7fbe6c000640 1 -- 172.21.15.27:0/1949872951 shutdown_connections 2024-04-16T05:56:16.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.835+0000 7fbe6c000640 1 --2- 172.21.15.27:0/1949872951 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fbe4806eb70 0x7fbe48071020 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:16.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.835+0000 7fbe6c000640 1 --2- 172.21.15.27:0/1949872951 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe64072310 0x7fbe64119a10 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:16.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.835+0000 7fbe6c000640 1 --2- 172.21.15.27:0/1949872951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbe64071a00 0x7fbe641194d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:16.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.835+0000 7fbe6c000640 1 -- 172.21.15.27:0/1949872951 >> 172.21.15.27:0/1949872951 conn(0x7fbe640fa1b0 msgr2=0x7fbe64106d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:16.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.835+0000 7fbe6c000640 1 -- 172.21.15.27:0/1949872951 shutdown_connections 2024-04-16T05:56:16.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:16.835+0000 7fbe6c000640 1 -- 172.21.15.27:0/1949872951 wait complete. 2024-04-16T05:56:16.853 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 32 2024-04-16T05:56:17.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:16 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/2977367302' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-16T05:56:17.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:16 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/2615735796' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-16T05:56:17.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:16 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/2977367302' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 29, "format": "json"}]: dispatch 2024-04-16T05:56:17.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:16 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/2615735796' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 30, "format": "json"}]: dispatch 2024-04-16T05:56:17.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.084+0000 7fd1a2e00640 1 -- 172.21.15.27:0/830213855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd19c06f870 msgr2=0x7fd19c06fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:17.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.084+0000 7fd1a2e00640 1 --2- 172.21.15.27:0/830213855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd19c06f870 0x7fd19c06fc40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd1900048d0 tx=0x7fd190009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:17.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.084+0000 7fd1a2e00640 1 -- 172.21.15.27:0/830213855 shutdown_connections 2024-04-16T05:56:17.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.084+0000 7fd1a2e00640 1 --2- 172.21.15.27:0/830213855 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd19c070180 0x7fd19c108310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.084+0000 7fd1a2e00640 1 --2- 172.21.15.27:0/830213855 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd19c06f870 0x7fd19c06fc40 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.084+0000 7fd1a2e00640 1 -- 172.21.15.27:0/830213855 >> 172.21.15.27:0/830213855 conn(0x7fd19c0f7b90 msgr2=0x7fd19c0f9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.084+0000 7fd1a2e00640 1 -- 172.21.15.27:0/830213855 shutdown_connections 2024-04-16T05:56:17.083 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.084+0000 7fd1a2e00640 1 -- 172.21.15.27:0/830213855 wait complete. 2024-04-16T05:56:17.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.085+0000 7fd1a2e00640 1 Processor -- start 2024-04-16T05:56:17.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.085+0000 7fd1a2e00640 1 -- start start 2024-04-16T05:56:17.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.086+0000 7fd1a2e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd19c06f870 0x7fd19c114d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:17.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.086+0000 7fd1a2e00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd19c070180 0x7fd19c115280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:17.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.086+0000 7fd1a2e00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c115880 con 0x7fd19c06f870 2024-04-16T05:56:17.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.086+0000 7fd1a2e00640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c1159f0 con 0x7fd19c070180 2024-04-16T05:56:17.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.086+0000 7fd1a1a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd19c06f870 0x7fd19c114d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.086+0000 7fd1a1a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd19c06f870 0x7fd19c114d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58808/0 (socket says 172.21.15.27:58808) 2024-04-16T05:56:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.086+0000 7fd1a1a00640 1 -- 172.21.15.27:0/699695376 learned_addr learned my addr 172.21.15.27:0/699695376 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.087+0000 7fd1a1a00640 1 -- 172.21.15.27:0/699695376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd19c070180 msgr2=0x7fd19c115280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.087+0000 7fd1a1a00640 1 --2- 172.21.15.27:0/699695376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd19c070180 0x7fd19c115280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:17.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.087+0000 7fd1a1a00640 1 -- 172.21.15.27:0/699695376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd190005040 con 0x7fd19c06f870 2024-04-16T05:56:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.087+0000 7fd1a1a00640 1 --2- 172.21.15.27:0/699695376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd19c06f870 0x7fd19c114d40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd1900047f0 tx=0x7fd190009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.087+0000 7fd199600640 1 -- 172.21.15.27:0/699695376 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1900094c0 con 0x7fd19c06f870 2024-04-16T05:56:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.087+0000 7fd199600640 1 -- 172.21.15.27:0/699695376 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd1900315f0 con 0x7fd19c06f870 2024-04-16T05:56:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.087+0000 7fd1a2e00640 1 -- 172.21.15.27:0/699695376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd19c10f020 con 0x7fd19c06f870 2024-04-16T05:56:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.087+0000 7fd199600640 1 -- 172.21.15.27:0/699695376 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1900435f0 con 0x7fd19c06f870 2024-04-16T05:56:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.087+0000 7fd1a2e00640 1 -- 172.21.15.27:0/699695376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd19c10f4e0 con 0x7fd19c06f870 2024-04-16T05:56:17.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.089+0000 7fd1a2e00640 1 -- 172.21.15.27:0/699695376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd19c070fb0 con 0x7fd19c06f870 2024-04-16T05:56:17.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.090+0000 7fd199600640 1 -- 172.21.15.27:0/699695376 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fd19002f070 con 0x7fd19c06f870 2024-04-16T05:56:17.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.090+0000 7fd199600640 1 --2- 172.21.15.27:0/699695376 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd18406eb70 0x7fd184071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:17.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.091+0000 7fd199600640 1 -- 172.21.15.27:0/699695376 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7fd1900b0870 con 0x7fd19c06f870 2024-04-16T05:56:17.088 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.091+0000 7fd1a1000640 1 --2- 172.21.15.27:0/699695376 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd18406eb70 0x7fd184071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:17.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.092+0000 7fd1a1000640 1 --2- 172.21.15.27:0/699695376 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd18406eb70 0x7fd184071020 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd19c110640 tx=0x7fd188007cb0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:17.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.095+0000 7fd199600640 1 -- 172.21.15.27:0/699695376 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fd1900794a0 con 0x7fd19c06f870 2024-04-16T05:56:17.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.379+0000 7fd1a2e00640 1 -- 172.21.15.27:0/699695376 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 32, "format": "json"} v 0) v1 -- 0x7fd19c10f810 con 0x7fd19c06f870 2024-04-16T05:56:17.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.381+0000 7fd199600640 1 -- 172.21.15.27:0/699695376 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 32, "format": "json"}]=0 dumped fsmap epoch 32 v36) v1 ==== 107+0+4347 (secure 0 0 0) 0x7fd19007cb60 con 0x7fd19c06f870 2024-04-16T05:56:17.378 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:17.379 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34432,"name":"cephfs.smithi027.ejoytk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6829/3219609959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3219609959},{"type":"v1","addr":"172.21.15.27:6829","nonce":3219609959}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":44337,"name":"cephfs.smithi122.lojjdp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6825/97103507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":97103507},{"type":"v1","addr":"172.21.15.122:6825","nonce":97103507}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T05:40:11.858650+0000","modified":"2024-04-16T05:51:44.023661+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34400},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34400":{"gid":34400,"name":"cephfs.smithi027.hkhpuq","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.27:6827/2831468256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2831468256},{"type":"v1","addr":"172.21.15.27:6827","nonce":2831468256}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T05:56:17.379 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 32 2024-04-16T05:56:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.384+0000 7fd1a2e00640 1 -- 172.21.15.27:0/699695376 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd18406eb70 msgr2=0x7fd184071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.384+0000 7fd1a2e00640 1 --2- 172.21.15.27:0/699695376 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd18406eb70 0x7fd184071020 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd19c110640 tx=0x7fd188007cb0 comp rx=0 tx=0).stop 2024-04-16T05:56:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.384+0000 7fd1a2e00640 1 -- 172.21.15.27:0/699695376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd19c06f870 msgr2=0x7fd19c114d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.384+0000 7fd1a2e00640 1 --2- 172.21.15.27:0/699695376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd19c06f870 0x7fd19c114d40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd1900047f0 tx=0x7fd190009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.384+0000 7fd1a2e00640 1 -- 172.21.15.27:0/699695376 shutdown_connections 2024-04-16T05:56:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.384+0000 7fd1a2e00640 1 --2- 172.21.15.27:0/699695376 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fd18406eb70 0x7fd184071020 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.384+0000 7fd1a2e00640 1 --2- 172.21.15.27:0/699695376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd19c070180 0x7fd19c115280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.384+0000 7fd1a2e00640 1 --2- 172.21.15.27:0/699695376 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd19c06f870 0x7fd19c114d40 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.384+0000 7fd1a2e00640 1 -- 172.21.15.27:0/699695376 >> 172.21.15.27:0/699695376 conn(0x7fd19c0f7b90 msgr2=0x7fd19c0f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.384+0000 7fd1a2e00640 1 -- 172.21.15.27:0/699695376 shutdown_connections 2024-04-16T05:56:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.384+0000 7fd1a2e00640 1 -- 172.21.15.27:0/699695376 wait complete. 2024-04-16T05:56:17.402 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 33 2024-04-16T05:56:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.633+0000 7f91b7800640 1 -- 172.21.15.27:0/1621358286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b01043c0 msgr2=0x7f91b0104790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.633+0000 7f91b7800640 1 --2- 172.21.15.27:0/1621358286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b01043c0 0x7f91b0104790 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f91a40048d0 tx=0x7f91a4009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.633+0000 7f91b7800640 1 -- 172.21.15.27:0/1621358286 shutdown_connections 2024-04-16T05:56:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.633+0000 7f91b7800640 1 --2- 172.21.15.27:0/1621358286 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91b0104d60 0x7f91b00fe3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:17.631 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.633+0000 7f91b7800640 1 --2- 172.21.15.27:0/1621358286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b01043c0 0x7f91b0104790 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:17.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.633+0000 7f91b7800640 1 -- 172.21.15.27:0/1621358286 >> 172.21.15.27:0/1621358286 conn(0x7f91b00f9fc0 msgr2=0x7f91b00fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:17.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.633+0000 7f91b7800640 1 -- 172.21.15.27:0/1621358286 shutdown_connections 2024-04-16T05:56:17.632 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.633+0000 7f91b7800640 1 -- 172.21.15.27:0/1621358286 wait complete. 2024-04-16T05:56:17.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.634+0000 7f91b7800640 1 Processor -- start 2024-04-16T05:56:17.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.634+0000 7f91b7800640 1 -- start start 2024-04-16T05:56:17.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.635+0000 7f91b7800640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91b01043c0 0x7f91b0114f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:17.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.635+0000 7f91b7800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b0104d60 0x7f91b0115480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:17.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.635+0000 7f91b7800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b0115a80 con 0x7f91b0104d60 2024-04-16T05:56:17.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.635+0000 7f91b7800640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b010f0d0 con 0x7f91b01043c0 2024-04-16T05:56:17.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.635+0000 7f91afe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b0104d60 0x7f91b0115480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:17.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.635+0000 7f91afe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b0104d60 0x7f91b0115480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58820/0 (socket says 172.21.15.27:58820) 2024-04-16T05:56:17.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.635+0000 7f91afe00640 1 -- 172.21.15.27:0/3859843260 learned_addr learned my addr 172.21.15.27:0/3859843260 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:17.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.636+0000 7f91b4e00640 1 --2- 172.21.15.27:0/3859843260 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91b01043c0 0x7f91b0114f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.636+0000 7f91afe00640 1 -- 172.21.15.27:0/3859843260 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91b01043c0 msgr2=0x7f91b0114f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.636+0000 7f91afe00640 1 --2- 172.21.15.27:0/3859843260 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91b01043c0 0x7f91b0114f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.636+0000 7f91afe00640 1 -- 172.21.15.27:0/3859843260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a4005040 con 0x7f91b0104d60 2024-04-16T05:56:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.636+0000 7f91afe00640 1 --2- 172.21.15.27:0/3859843260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b0104d60 0x7f91b0115480 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f919c00c790 tx=0x7f919c0043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.637+0000 7f91acc00640 1 -- 172.21.15.27:0/3859843260 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f919c00a7a0 con 0x7f91b0104d60 2024-04-16T05:56:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.637+0000 7f91acc00640 1 -- 172.21.15.27:0/3859843260 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f919c00add0 con 0x7f91b0104d60 2024-04-16T05:56:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.637+0000 7f91b7800640 1 -- 172.21.15.27:0/3859843260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b010f3b0 con 0x7f91b0104d60 2024-04-16T05:56:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.637+0000 7f91acc00640 1 -- 172.21.15.27:0/3859843260 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f919c0115e0 con 0x7f91b0104d60 2024-04-16T05:56:17.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.637+0000 7f91b7800640 1 -- 172.21.15.27:0/3859843260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b010f8d0 con 0x7f91b0104d60 2024-04-16T05:56:17.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.638+0000 7f91b7800640 1 -- 172.21.15.27:0/3859843260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b00ffb30 con 0x7f91b0104d60 2024-04-16T05:56:17.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.639+0000 7f91acc00640 1 -- 172.21.15.27:0/3859843260 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f919c012400 con 0x7f91b0104d60 2024-04-16T05:56:17.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.640+0000 7f91acc00640 1 --2- 172.21.15.27:0/3859843260 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f919806eb70 0x7f9198071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:17.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.640+0000 7f91acc00640 1 -- 172.21.15.27:0/3859843260 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f919c08eae0 con 0x7f91b0104d60 2024-04-16T05:56:17.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.640+0000 7f91b4e00640 1 --2- 172.21.15.27:0/3859843260 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f919806eb70 0x7f9198071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:17.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.641+0000 7f91b4e00640 1 --2- 172.21.15.27:0/3859843260 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f919806eb70 0x7f9198071020 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f91a40048a0 tx=0x7f91a4009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:17.642 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.645+0000 7f91acc00640 1 -- 172.21.15.27:0/3859843260 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f919c0577b0 con 0x7f91b0104d60 2024-04-16T05:56:17.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.893+0000 7f91b7800640 1 -- 172.21.15.27:0/3859843260 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 33, "format": "json"} v 0) v1 -- 0x7f91b010b550 con 0x7f91b0104d60 2024-04-16T05:56:17.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.894+0000 7f91acc00640 1 -- 172.21.15.27:0/3859843260 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 33, "format": "json"}]=0 dumped fsmap epoch 33 v36) v1 ==== 107+0+4351 (secure 0 0 0) 0x7f919c05ae70 con 0x7f91b0104d60 2024-04-16T05:56:17.892 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:17.892 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34432,"name":"cephfs.smithi027.ejoytk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6829/3219609959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3219609959},{"type":"v1","addr":"172.21.15.27:6829","nonce":3219609959}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":44337,"name":"cephfs.smithi122.lojjdp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6825/97103507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":97103507},{"type":"v1","addr":"172.21.15.122:6825","nonce":97103507}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T05:40:11.858650+0000","modified":"2024-04-16T05:51:44.051526+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34400},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34400":{"gid":34400,"name":"cephfs.smithi027.hkhpuq","rank":0,"incarnation":32,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.27:6827/2831468256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2831468256},{"type":"v1","addr":"172.21.15.27:6827","nonce":2831468256}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T05:56:17.893 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 33 2024-04-16T05:56:17.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.897+0000 7f91b7800640 1 -- 172.21.15.27:0/3859843260 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f919806eb70 msgr2=0x7f9198071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:17.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.897+0000 7f91b7800640 1 --2- 172.21.15.27:0/3859843260 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f919806eb70 0x7f9198071020 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f91a40048a0 tx=0x7f91a4009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:17.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.897+0000 7f91b7800640 1 -- 172.21.15.27:0/3859843260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b0104d60 msgr2=0x7f91b0115480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:17.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.897+0000 7f91b7800640 1 --2- 172.21.15.27:0/3859843260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b0104d60 0x7f91b0115480 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f919c00c790 tx=0x7f919c0043c0 comp rx=0 tx=0).stop 2024-04-16T05:56:17.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.897+0000 7f91b7800640 1 -- 172.21.15.27:0/3859843260 shutdown_connections 2024-04-16T05:56:17.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.897+0000 7f91b7800640 1 --2- 172.21.15.27:0/3859843260 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f919806eb70 0x7f9198071020 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:17.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.897+0000 7f91b7800640 1 --2- 172.21.15.27:0/3859843260 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91b0104d60 0x7f91b0115480 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:17.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.897+0000 7f91b7800640 1 --2- 172.21.15.27:0/3859843260 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f91b01043c0 0x7f91b0114f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:17.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.897+0000 7f91b7800640 1 -- 172.21.15.27:0/3859843260 >> 172.21.15.27:0/3859843260 conn(0x7f91b00f9fc0 msgr2=0x7f91b00faf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:17.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.897+0000 7f91b7800640 1 -- 172.21.15.27:0/3859843260 shutdown_connections 2024-04-16T05:56:17.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:17.898+0000 7f91b7800640 1 -- 172.21.15.27:0/3859843260 wait complete. 2024-04-16T05:56:17.914 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 34 2024-04-16T05:56:17.999 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:17 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1949872951' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-16T05:56:18.000 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:17 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/699695376' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-16T05:56:18.000 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:17 smithi027 ceph-mon[93635]: pgmap v366: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:18.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:17 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1949872951' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 31, "format": "json"}]: dispatch 2024-04-16T05:56:18.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:17 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/699695376' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 32, "format": "json"}]: dispatch 2024-04-16T05:56:18.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:17 smithi122 ceph-mon[80930]: pgmap v366: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:18.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.135+0000 7effabe00640 1 -- 172.21.15.27:0/739581290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effac1029b0 msgr2=0x7effac0fc040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:18.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.135+0000 7effabe00640 1 --2- 172.21.15.27:0/739581290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effac1029b0 0x7effac0fc040 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7eff980048d0 tx=0x7eff98009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:18.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.135+0000 7effabe00640 1 -- 172.21.15.27:0/739581290 shutdown_connections 2024-04-16T05:56:18.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.135+0000 7effabe00640 1 --2- 172.21.15.27:0/739581290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effac1029b0 0x7effac0fc040 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:18.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.135+0000 7effabe00640 1 --2- 172.21.15.27:0/739581290 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7effac102010 0x7effac1023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:18.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.135+0000 7effabe00640 1 -- 172.21.15.27:0/739581290 >> 172.21.15.27:0/739581290 conn(0x7effac0f7b80 msgr2=0x7effac0f9f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:18.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.135+0000 7effabe00640 1 -- 172.21.15.27:0/739581290 shutdown_connections 2024-04-16T05:56:18.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.135+0000 7effabe00640 1 -- 172.21.15.27:0/739581290 wait complete. 2024-04-16T05:56:18.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.136+0000 7effabe00640 1 Processor -- start 2024-04-16T05:56:18.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.136+0000 7effabe00640 1 -- start start 2024-04-16T05:56:18.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.136+0000 7effabe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effac102010 0x7effac114d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:18.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.136+0000 7effabe00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7effac1029b0 0x7effac115260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:18.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.136+0000 7effabe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effac115860 con 0x7effac102010 2024-04-16T05:56:18.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.136+0000 7effabe00640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effac10eeb0 con 0x7effac1029b0 2024-04-16T05:56:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.136+0000 7effaaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effac102010 0x7effac114d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.136+0000 7effaaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effac102010 0x7effac114d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58838/0 (socket says 172.21.15.27:58838) 2024-04-16T05:56:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.136+0000 7effaaa00640 1 -- 172.21.15.27:0/2500235724 learned_addr learned my addr 172.21.15.27:0/2500235724 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.137+0000 7effaa000640 1 --2- 172.21.15.27:0/2500235724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7effac1029b0 0x7effac115260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.137+0000 7effaaa00640 1 -- 172.21.15.27:0/2500235724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7effac1029b0 msgr2=0x7effac115260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.137+0000 7effaaa00640 1 --2- 172.21.15.27:0/2500235724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7effac1029b0 0x7effac115260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.137+0000 7effaaa00640 1 -- 172.21.15.27:0/2500235724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff98005040 con 0x7effac102010 2024-04-16T05:56:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.137+0000 7effaaa00640 1 --2- 172.21.15.27:0/2500235724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effac102010 0x7effac114d20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7eff9c002a50 tx=0x7eff9c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.137+0000 7effa2a00640 1 -- 172.21.15.27:0/2500235724 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff9c007c30 con 0x7effac102010 2024-04-16T05:56:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.137+0000 7effa2a00640 1 -- 172.21.15.27:0/2500235724 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff9c00f040 con 0x7effac102010 2024-04-16T05:56:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.137+0000 7effabe00640 1 -- 172.21.15.27:0/2500235724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effac10f190 con 0x7effac102010 2024-04-16T05:56:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.138+0000 7effa2a00640 1 -- 172.21.15.27:0/2500235724 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff9c00db50 con 0x7effac102010 2024-04-16T05:56:18.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.138+0000 7effabe00640 1 -- 172.21.15.27:0/2500235724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effac10f6b0 con 0x7effac102010 2024-04-16T05:56:18.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.139+0000 7effabe00640 1 -- 172.21.15.27:0/2500235724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effac0fd780 con 0x7effac102010 2024-04-16T05:56:18.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.140+0000 7effa2a00640 1 -- 172.21.15.27:0/2500235724 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7eff9c00dcf0 con 0x7effac102010 2024-04-16T05:56:18.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.141+0000 7effa2a00640 1 --2- 172.21.15.27:0/2500235724 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7eff9006eb70 0x7eff90071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:18.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.142+0000 7effa2a00640 1 -- 172.21.15.27:0/2500235724 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7eff9c08e2e0 con 0x7effac102010 2024-04-16T05:56:18.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.142+0000 7effaa000640 1 --2- 172.21.15.27:0/2500235724 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7eff9006eb70 0x7eff90071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:18.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.143+0000 7effaa000640 1 --2- 172.21.15.27:0/2500235724 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7eff9006eb70 0x7eff90071020 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7eff98000c00 tx=0x7eff98009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:18.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.146+0000 7effa2a00640 1 -- 172.21.15.27:0/2500235724 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7eff9c056f10 con 0x7effac102010 2024-04-16T05:56:18.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.394+0000 7effabe00640 1 -- 172.21.15.27:0/2500235724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "epoch": 34, "format": "json"} v 0) v1 -- 0x7effac110420 con 0x7effac102010 2024-04-16T05:56:18.393 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.396+0000 7effa2a00640 1 -- 172.21.15.27:0/2500235724 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "epoch": 34, "format": "json"}]=0 dumped fsmap epoch 34 v36) v1 ==== 107+0+4348 (secure 0 0 0) 0x7eff9c05a5d0 con 0x7effac102010 2024-04-16T05:56:18.394 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:18.394 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":34432,"name":"cephfs.smithi027.ejoytk","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6829/3219609959","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":3219609959},{"type":"v1","addr":"172.21.15.27:6829","nonce":3219609959}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24},{"gid":44337,"name":"cephfs.smithi122.lojjdp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6825/97103507","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":97103507},{"type":"v1","addr":"172.21.15.122:6825","nonce":97103507}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-16T05:40:11.858650+0000","modified":"2024-04-16T05:51:45.036713+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":88,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":34400},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34400":{"gid":34400,"name":"cephfs.smithi027.hkhpuq","rank":0,"incarnation":32,"state":"up:rejoin","state_seq":22,"addr":"172.21.15.27:6827/2831468256","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":2831468256},{"type":"v1","addr":"172.21.15.27:6827","nonce":2831468256}]},"join_fscid":1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":1}]} 2024-04-16T05:56:18.394 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 34 2024-04-16T05:56:18.396 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.398+0000 7effabe00640 1 -- 172.21.15.27:0/2500235724 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7eff9006eb70 msgr2=0x7eff90071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:18.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.398+0000 7effabe00640 1 --2- 172.21.15.27:0/2500235724 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7eff9006eb70 0x7eff90071020 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7eff98000c00 tx=0x7eff98009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:18.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.399+0000 7effabe00640 1 -- 172.21.15.27:0/2500235724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effac102010 msgr2=0x7effac114d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:18.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.399+0000 7effabe00640 1 --2- 172.21.15.27:0/2500235724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effac102010 0x7effac114d20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7eff9c002a50 tx=0x7eff9c004600 comp rx=0 tx=0).stop 2024-04-16T05:56:18.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.399+0000 7effabe00640 1 -- 172.21.15.27:0/2500235724 shutdown_connections 2024-04-16T05:56:18.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.399+0000 7effabe00640 1 --2- 172.21.15.27:0/2500235724 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7eff9006eb70 0x7eff90071020 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:18.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.399+0000 7effabe00640 1 --2- 172.21.15.27:0/2500235724 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7effac1029b0 0x7effac115260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:18.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.399+0000 7effabe00640 1 --2- 172.21.15.27:0/2500235724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effac102010 0x7effac114d20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:18.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.399+0000 7effabe00640 1 -- 172.21.15.27:0/2500235724 >> 172.21.15.27:0/2500235724 conn(0x7effac0f7b80 msgr2=0x7effac071180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:18.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.399+0000 7effabe00640 1 -- 172.21.15.27:0/2500235724 shutdown_connections 2024-04-16T05:56:18.397 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.399+0000 7effabe00640 1 -- 172.21.15.27:0/2500235724 wait complete. 2024-04-16T05:56:18.418 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-16T05:56:18.433 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-16T05:56:18.433 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-16T05:56:18.433 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T05:56:18.495 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-16T05:56:18.495 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T05:56:18.553 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T05:56:18.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.786+0000 7f8fcfe00640 1 -- 172.21.15.27:0/2826867380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd0102630 msgr2=0x7f8fd0102aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:18.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.786+0000 7f8fcfe00640 1 --2- 172.21.15.27:0/2826867380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd0102630 0x7f8fd0102aa0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc0048d0 tx=0x7f8fbc009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:18.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.786+0000 7f8fcfe00640 1 -- 172.21.15.27:0/2826867380 shutdown_connections 2024-04-16T05:56:18.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.786+0000 7f8fcfe00640 1 --2- 172.21.15.27:0/2826867380 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd0102630 0x7f8fd0102aa0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:18.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.786+0000 7f8fcfe00640 1 --2- 172.21.15.27:0/2826867380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fd0107560 0x7f8fd0107930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:18.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.786+0000 7f8fcfe00640 1 -- 172.21.15.27:0/2826867380 >> 172.21.15.27:0/2826867380 conn(0x7f8fd0071e50 msgr2=0x7f8fd0074260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:18.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.786+0000 7f8fcfe00640 1 -- 172.21.15.27:0/2826867380 shutdown_connections 2024-04-16T05:56:18.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.786+0000 7f8fcfe00640 1 -- 172.21.15.27:0/2826867380 wait complete. 2024-04-16T05:56:18.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.787+0000 7f8fcfe00640 1 Processor -- start 2024-04-16T05:56:18.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.787+0000 7f8fcfe00640 1 -- start start 2024-04-16T05:56:18.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.787+0000 7f8fcfe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd0102630 0x7f8fd01190f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:18.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.787+0000 7f8fcfe00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fd0107560 0x7f8fd0119630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:18.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.788+0000 7f8fcfe00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd0119c30 con 0x7f8fd0102630 2024-04-16T05:56:18.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.788+0000 7f8fcfe00640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd0113280 con 0x7f8fd0107560 2024-04-16T05:56:18.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.788+0000 7f8fcea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd0102630 0x7f8fd01190f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:18.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.788+0000 7f8fcea00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd0102630 0x7f8fd01190f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58848/0 (socket says 172.21.15.27:58848) 2024-04-16T05:56:18.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.788+0000 7f8fcea00640 1 -- 172.21.15.27:0/382879530 learned_addr learned my addr 172.21.15.27:0/382879530 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:18.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.788+0000 7f8fcea00640 1 -- 172.21.15.27:0/382879530 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fd0107560 msgr2=0x7f8fd0119630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:18.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.788+0000 7f8fcea00640 1 --2- 172.21.15.27:0/382879530 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fd0107560 0x7f8fd0119630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:18.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.788+0000 7f8fcea00640 1 -- 172.21.15.27:0/382879530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fbc005040 con 0x7f8fd0102630 2024-04-16T05:56:18.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.789+0000 7f8fcea00640 1 --2- 172.21.15.27:0/382879530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd0102630 0x7f8fd01190f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0002840 tx=0x7f8fc0004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:18.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.789+0000 7f8fc6a00640 1 -- 172.21.15.27:0/382879530 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fc0007c30 con 0x7f8fd0102630 2024-04-16T05:56:18.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.789+0000 7f8fc6a00640 1 -- 172.21.15.27:0/382879530 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8fc000f040 con 0x7f8fd0102630 2024-04-16T05:56:18.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.789+0000 7f8fcfe00640 1 -- 172.21.15.27:0/382879530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd0113560 con 0x7f8fd0102630 2024-04-16T05:56:18.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.789+0000 7f8fc6a00640 1 -- 172.21.15.27:0/382879530 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fc0009590 con 0x7f8fd0102630 2024-04-16T05:56:18.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.789+0000 7f8fcfe00640 1 -- 172.21.15.27:0/382879530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd0113a80 con 0x7f8fd0102630 2024-04-16T05:56:18.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.790+0000 7f8fcfe00640 1 -- 172.21.15.27:0/382879530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd0103d20 con 0x7f8fd0102630 2024-04-16T05:56:18.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.791+0000 7f8fc6a00640 1 -- 172.21.15.27:0/382879530 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f8fc000d6b0 con 0x7f8fd0102630 2024-04-16T05:56:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.792+0000 7f8fc6a00640 1 --2- 172.21.15.27:0/382879530 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8fb406eb70 0x7f8fb4071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.792+0000 7f8fc6a00640 1 -- 172.21.15.27:0/382879530 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f8fc008ea50 con 0x7f8fd0102630 2024-04-16T05:56:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.792+0000 7f8fce000640 1 --2- 172.21.15.27:0/382879530 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8fb406eb70 0x7f8fb4071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.793+0000 7f8fce000640 1 --2- 172.21.15.27:0/382879530 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8fb406eb70 0x7f8fb4071020 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc0043d0 tx=0x7f8fbc009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:18.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:18.797+0000 7f8fc6a00640 1 -- 172.21.15.27:0/382879530 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f8fc0057560 con 0x7f8fd0102630 2024-04-16T05:56:19.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.005+0000 7f8fcfe00640 1 -- 172.21.15.27:0/382879530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fd010f7d0 con 0x7f8fd0102630 2024-04-16T05:56:19.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.009+0000 7f8fc6a00640 1 -- 172.21.15.27:0/382879530 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+17388 (secure 0 0 0) 0x7f8fc005ac20 con 0x7f8fd0102630 2024-04-16T05:56:19.007 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:19.007 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":88,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","created":"2024-04-16T05:31:01.142297+0000","modified":"2024-04-16T05:51:44.010556+0000","last_up_change":"2024-04-16T05:50:07.322376+0000","last_in_change":"2024-04-16T05:38:07.718747+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T05:36:52.435427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-16T05:40:10.776546+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-16T05:40:11.582658+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4dd7e294-f625-4492-8bd3-18beab34b6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":56,"up_thru":82,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1955571017},{"type":"v1","addr":"172.21.15.27:6803","nonce":1955571017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1955571017},{"type":"v1","addr":"172.21.15.27:6805","nonce":1955571017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1955571017},{"type":"v1","addr":"172.21.15.27:6809","nonce":1955571017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1955571017},{"type":"v1","addr":"172.21.15.27:6807","nonce":1955571017}]},"public_addr":"172.21.15.27:6803/1955571017","cluster_addr":"172.21.15.27:6805/1955571017","heartbeat_back_addr":"172.21.15.27:6809/1955571017","heartbeat_front_addr":"172.21.15.27:6807/1955571017","state":["exists","up"]},{"osd":1,"uuid":"abec61aa-7a28-473b-b9d2-04e8d7de8a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":58,"up_from":62,"up_thru":82,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2379867635},{"type":"v1","addr":"172.21.15.27:6811","nonce":2379867635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2379867635},{"type":"v1","addr":"172.21.15.27:6813","nonce":2379867635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2379867635},{"type":"v1","addr":"172.21.15.27:6817","nonce":2379867635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2379867635},{"type":"v1","addr":"172.21.15.27:6815","nonce":2379867635}]},"public_addr":"172.21.15.27:6811/2379867635","cluster_addr":"172.21.15.27:6813/2379867635","heartbeat_back_addr":"172.21.15.27:6817/2379867635","heartbeat_front_addr":"172.21.15.27:6815/2379867635","state":["exists","up"]},{"osd":2,"uuid":"e50778e4-bf1f-45eb-aeab-05e8dad8fa89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":63,"up_from":67,"up_thru":82,"down_at":64,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1424147666},{"type":"v1","addr":"172.21.15.27:6819","nonce":1424147666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1424147666},{"type":"v1","addr":"172.21.15.27:6821","nonce":1424147666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1424147666},{"type":"v1","addr":"172.21.15.27:6825","nonce":1424147666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1424147666},{"type":"v1","addr":"172.21.15.27:6823","nonce":1424147666}]},"public_addr":"172.21.15.27:6819/1424147666","cluster_addr":"172.21.15.27:6821/1424147666","heartbeat_back_addr":"172.21.15.27:6825/1424147666","heartbeat_front_addr":"172.21.15.27:6823/1424147666","state":["exists","up"]},{"osd":3,"uuid":"8f55dd70-a9d7-4bc1-a79c-028e66571137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":68,"up_from":72,"up_thru":82,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3174341439},{"type":"v1","addr":"172.21.15.122:6801","nonce":3174341439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3174341439},{"type":"v1","addr":"172.21.15.122:6803","nonce":3174341439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3174341439},{"type":"v1","addr":"172.21.15.122:6807","nonce":3174341439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3174341439},{"type":"v1","addr":"172.21.15.122:6805","nonce":3174341439}]},"public_addr":"172.21.15.122:6801/3174341439","cluster_addr":"172.21.15.122:6803/3174341439","heartbeat_back_addr":"172.21.15.122:6807/3174341439","heartbeat_front_addr":"172.21.15.122:6805/3174341439","state":["exists","up"]},{"osd":4,"uuid":"ba8734db-cb7a-4242-b354-8ff1f524f25a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":73,"up_from":77,"up_thru":82,"down_at":74,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2308306674},{"type":"v1","addr":"172.21.15.122:6809","nonce":2308306674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2308306674},{"type":"v1","addr":"172.21.15.122:6811","nonce":2308306674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2308306674},{"type":"v1","addr":"172.21.15.122:6815","nonce":2308306674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2308306674},{"type":"v1","addr":"172.21.15.122:6813","nonce":2308306674}]},"public_addr":"172.21.15.122:6809/2308306674","cluster_addr":"172.21.15.122:6811/2308306674","heartbeat_back_addr":"172.21.15.122:6815/2308306674","heartbeat_front_addr":"172.21.15.122:6813/2308306674","state":["exists","up"]},{"osd":5,"uuid":"6e2753a7-d194-4103-b586-bfef8cbefa2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":78,"up_from":82,"up_thru":82,"down_at":79,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":3634763653},{"type":"v1","addr":"172.21.15.122:6817","nonce":3634763653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":3634763653},{"type":"v1","addr":"172.21.15.122:6819","nonce":3634763653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":3634763653},{"type":"v1","addr":"172.21.15.122:6823","nonce":3634763653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":3634763653},{"type":"v1","addr":"172.21.15.122:6821","nonce":3634763653}]},"public_addr":"172.21.15.122:6817/3634763653","cluster_addr":"172.21.15.122:6819/3634763653","heartbeat_back_addr":"172.21.15.122:6823/3634763653","heartbeat_front_addr":"172.21.15.122:6821/3634763653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:35:28.179537+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:36:08.890364+0000","dead_epoch":58},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:36:48.694394+0000","dead_epoch":63},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:37:21.969883+0000","dead_epoch":68},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:37:55.840648+0000","dead_epoch":73},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:38:29.596986+0000","dead_epoch":78}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":3,"to":2}]},{"pgid":"2.a","mappings":[{"from":3,"to":0}]},{"pgid":"2.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.6","mappings":[{"from":1,"to":5}]},{"pgid":"3.b","mappings":[{"from":1,"to":2}]},{"pgid":"3.d","mappings":[{"from":3,"to":0}]},{"pgid":"3.15","mappings":[{"from":3,"to":2},{"from":4,"to":5}]},{"pgid":"3.1c","mappings":[{"from":4,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.122:6825/194115669":"2024-04-17T05:51:15.646390+0000","172.21.15.122:6824/194115669":"2024-04-17T05:51:15.646390+0000","172.21.15.27:6829/4115272713":"2024-04-17T05:50:45.882787+0000","172.21.15.122:6827/1078307975":"2024-04-17T05:51:44.010470+0000","172.21.15.27:6828/4115272713":"2024-04-17T05:50:45.882787+0000","172.21.15.27:0/3779871471":"2024-04-17T05:45:28.872665+0000","172.21.15.27:0/3077390652":"2024-04-17T05:43:00.762041+0000","172.21.15.27:6800/2779081505":"2024-04-17T05:32:17.529651+0000","172.21.15.27:6801/2173136120":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/3170976796":"2024-04-17T05:31:34.994394+0000","172.21.15.27:6800/2203377971":"2024-04-17T05:43:00.762041+0000","172.21.15.122:6826/1078307975":"2024-04-17T05:51:44.010470+0000","172.21.15.122:0/1743593984":"2024-04-17T05:44:23.662611+0000","172.21.15.27:6801/2779081505":"2024-04-17T05:32:17.529651+0000","172.21.15.27:0/4258676557":"2024-04-17T05:32:17.529651+0000","172.21.15.122:0/575464263":"2024-04-17T05:44:23.662611+0000","172.21.15.27:0/2398184523":"2024-04-17T05:45:28.872665+0000","172.21.15.27:0/4061328916":"2024-04-17T05:43:00.762041+0000","172.21.15.27:0/99903288":"2024-04-17T05:43:00.762041+0000","172.21.15.27:6826/1980301393":"2024-04-17T05:40:29.686229+0000","172.21.15.27:6800/2700787222":"2024-04-17T05:33:49.465983+0000","172.21.15.27:6800/1090905394":"2024-04-17T05:45:28.872665+0000","172.21.15.122:0/2486581297":"2024-04-17T05:44:23.662611+0000","172.21.15.27:6801/2700787222":"2024-04-17T05:33:49.465983+0000","172.21.15.27:6827/1980301393":"2024-04-17T05:40:29.686229+0000","172.21.15.27:0/3086153578":"2024-04-17T05:32:17.529651+0000","172.21.15.122:6828/871482489":"2024-04-17T05:44:23.662611+0000","172.21.15.27:0/1860000032":"2024-04-17T05:33:49.465983+0000","172.21.15.122:6829/871482489":"2024-04-17T05:44:23.662611+0000","172.21.15.27:0/602364106":"2024-04-17T05:33:49.465983+0000","172.21.15.122:0/915233781":"2024-04-17T05:44:23.662611+0000","172.21.15.27:0/3775595668":"2024-04-17T05:32:17.529651+0000","172.21.15.122:0/2407794149":"2024-04-17T05:44:23.662611+0000","172.21.15.27:0/1372853588":"2024-04-17T05:45:28.872665+0000","172.21.15.27:0/2242461930":"2024-04-17T05:33:49.465983+0000","172.21.15.27:0/2783469284":"2024-04-17T05:45:28.872665+0000","172.21.15.27:6800/2173136120":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/3244097830":"2024-04-17T05:45:28.872665+0000","172.21.15.27:6801/2203377971":"2024-04-17T05:43:00.762041+0000","172.21.15.122:0/3869652783":"2024-04-17T05:44:23.662611+0000","172.21.15.27:0/1691369286":"2024-04-17T05:45:28.872665+0000","172.21.15.27:0/1435069943":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/3713909498":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/2938698524":"2024-04-17T05:43:00.762041+0000","172.21.15.27:6801/1090905394":"2024-04-17T05:45:28.872665+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T05:56:19.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.012+0000 7f8fcfe00640 1 -- 172.21.15.27:0/382879530 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8fb406eb70 msgr2=0x7f8fb4071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:19.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.012+0000 7f8fcfe00640 1 --2- 172.21.15.27:0/382879530 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8fb406eb70 0x7f8fb4071020 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc0043d0 tx=0x7f8fbc009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:19.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.012+0000 7f8fcfe00640 1 -- 172.21.15.27:0/382879530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd0102630 msgr2=0x7f8fd01190f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:19.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.012+0000 7f8fcfe00640 1 --2- 172.21.15.27:0/382879530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd0102630 0x7f8fd01190f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0002840 tx=0x7f8fc0004570 comp rx=0 tx=0).stop 2024-04-16T05:56:19.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.012+0000 7f8fcfe00640 1 -- 172.21.15.27:0/382879530 shutdown_connections 2024-04-16T05:56:19.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.012+0000 7f8fcfe00640 1 --2- 172.21.15.27:0/382879530 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f8fb406eb70 0x7f8fb4071020 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:19.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.012+0000 7f8fcfe00640 1 --2- 172.21.15.27:0/382879530 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8fd0107560 0x7f8fd0119630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:19.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.012+0000 7f8fcfe00640 1 --2- 172.21.15.27:0/382879530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8fd0102630 0x7f8fd01190f0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:19.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.012+0000 7f8fcfe00640 1 -- 172.21.15.27:0/382879530 >> 172.21.15.27:0/382879530 conn(0x7f8fd0071e50 msgr2=0x7f8fd0104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:19.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.012+0000 7f8fcfe00640 1 -- 172.21.15.27:0/382879530 shutdown_connections 2024-04-16T05:56:19.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.012+0000 7f8fcfe00640 1 -- 172.21.15.27:0/382879530 wait complete. 2024-04-16T05:56:19.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:18 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/3859843260' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-16T05:56:19.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:18 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/2500235724' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-16T05:56:19.027 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:18 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/3859843260' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 33, "format": "json"}]: dispatch 2024-04-16T05:56:19.028 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:18 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/2500235724' entity='client.admin' cmd=[{"prefix": "fs dump", "epoch": 34, "format": "json"}]: dispatch 2024-04-16T05:56:19.032 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T05:36:52.435427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-16T05:40:10.776546+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-16T05:40:11.582658+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T05:56:19.033 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T05:56:19.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.247+0000 7fe4ac400640 1 -- 172.21.15.27:0/3388357656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a40714a0 msgr2=0x7fe4a406f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:19.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.247+0000 7fe4ac400640 1 --2- 172.21.15.27:0/3388357656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a40714a0 0x7fe4a406f870 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe4980048d0 tx=0x7fe498009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:19.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.247+0000 7fe4ac400640 1 -- 172.21.15.27:0/3388357656 shutdown_connections 2024-04-16T05:56:19.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.247+0000 7fe4ac400640 1 --2- 172.21.15.27:0/3388357656 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4a406fdb0 0x7fe4a4070220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:19.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.247+0000 7fe4ac400640 1 --2- 172.21.15.27:0/3388357656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a40714a0 0x7fe4a406f870 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:19.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.247+0000 7fe4ac400640 1 -- 172.21.15.27:0/3388357656 >> 172.21.15.27:0/3388357656 conn(0x7fe4a40f7fb0 msgr2=0x7fe4a40fa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:19.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.248+0000 7fe4ac400640 1 -- 172.21.15.27:0/3388357656 shutdown_connections 2024-04-16T05:56:19.246 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.248+0000 7fe4ac400640 1 -- 172.21.15.27:0/3388357656 wait complete. 2024-04-16T05:56:19.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.248+0000 7fe4ac400640 1 Processor -- start 2024-04-16T05:56:19.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.249+0000 7fe4ac400640 1 -- start start 2024-04-16T05:56:19.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.249+0000 7fe4ac400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a406fdb0 0x7fe4a4119560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:19.247 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.249+0000 7fe4ac400640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4a40714a0 0x7fe4a4119aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:19.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.249+0000 7fe4ac400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a411a0f0 con 0x7fe4a406fdb0 2024-04-16T05:56:19.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.249+0000 7fe4ac400640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a4113710 con 0x7fe4a40714a0 2024-04-16T05:56:19.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.249+0000 7fe4a9a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a406fdb0 0x7fe4a4119560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:19.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.249+0000 7fe4a9a00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a406fdb0 0x7fe4a4119560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58880/0 (socket says 172.21.15.27:58880) 2024-04-16T05:56:19.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.249+0000 7fe4a9a00640 1 -- 172.21.15.27:0/85486453 learned_addr learned my addr 172.21.15.27:0/85486453 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:19.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.250+0000 7fe4a9000640 1 --2- 172.21.15.27:0/85486453 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4a40714a0 0x7fe4a4119aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:19.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.250+0000 7fe4a9a00640 1 -- 172.21.15.27:0/85486453 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4a40714a0 msgr2=0x7fe4a4119aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:19.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.250+0000 7fe4a9a00640 1 --2- 172.21.15.27:0/85486453 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4a40714a0 0x7fe4a4119aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:19.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.250+0000 7fe4a9a00640 1 -- 172.21.15.27:0/85486453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe498005040 con 0x7fe4a406fdb0 2024-04-16T05:56:19.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.250+0000 7fe4a9a00640 1 --2- 172.21.15.27:0/85486453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a406fdb0 0x7fe4a4119560 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe4980048a0 tx=0x7fe498009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:19.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.251+0000 7fe4a1600640 1 -- 172.21.15.27:0/85486453 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe4980092e0 con 0x7fe4a406fdb0 2024-04-16T05:56:19.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.251+0000 7fe4a1600640 1 -- 172.21.15.27:0/85486453 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe498009440 con 0x7fe4a406fdb0 2024-04-16T05:56:19.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.251+0000 7fe4ac400640 1 -- 172.21.15.27:0/85486453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a4113990 con 0x7fe4a406fdb0 2024-04-16T05:56:19.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.251+0000 7fe4a1600640 1 -- 172.21.15.27:0/85486453 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe498043640 con 0x7fe4a406fdb0 2024-04-16T05:56:19.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.251+0000 7fe4ac400640 1 -- 172.21.15.27:0/85486453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a4113eb0 con 0x7fe4a406fdb0 2024-04-16T05:56:19.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.252+0000 7fe4ac400640 1 -- 172.21.15.27:0/85486453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a4070fb0 con 0x7fe4a406fdb0 2024-04-16T05:56:19.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.253+0000 7fe4a1600640 1 -- 172.21.15.27:0/85486453 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fe49803b420 con 0x7fe4a406fdb0 2024-04-16T05:56:19.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.254+0000 7fe4a1600640 1 --2- 172.21.15.27:0/85486453 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe48c06eb70 0x7fe48c071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:19.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.254+0000 7fe4a1600640 1 -- 172.21.15.27:0/85486453 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7fe4980b1840 con 0x7fe4a406fdb0 2024-04-16T05:56:19.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.255+0000 7fe4a9000640 1 --2- 172.21.15.27:0/85486453 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe48c06eb70 0x7fe48c071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:19.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.255+0000 7fe4a9000640 1 --2- 172.21.15.27:0/85486453 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe48c06eb70 0x7fe48c071020 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe490007fd0 tx=0x7fe490009e90 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:19.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.260+0000 7fe4a1600640 1 -- 172.21.15.27:0/85486453 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fe49807a450 con 0x7fe4a406fdb0 2024-04-16T05:56:19.464 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.466+0000 7fe4ac400640 1 -- 172.21.15.27:0/85486453 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe4a410fb20 con 0x7fe4a406fdb0 2024-04-16T05:56:19.465 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.467+0000 7fe4a1600640 1 -- 172.21.15.27:0/85486453 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe49807db10 con 0x7fe4a406fdb0 2024-04-16T05:56:19.465 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-16T05:56:19.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.470+0000 7fe4ac400640 1 -- 172.21.15.27:0/85486453 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe48c06eb70 msgr2=0x7fe48c071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:19.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.470+0000 7fe4ac400640 1 --2- 172.21.15.27:0/85486453 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe48c06eb70 0x7fe48c071020 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe490007fd0 tx=0x7fe490009e90 comp rx=0 tx=0).stop 2024-04-16T05:56:19.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.470+0000 7fe4ac400640 1 -- 172.21.15.27:0/85486453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a406fdb0 msgr2=0x7fe4a4119560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:19.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.470+0000 7fe4ac400640 1 --2- 172.21.15.27:0/85486453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a406fdb0 0x7fe4a4119560 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe4980048a0 tx=0x7fe498009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:19.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.470+0000 7fe4ac400640 1 -- 172.21.15.27:0/85486453 shutdown_connections 2024-04-16T05:56:19.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.470+0000 7fe4ac400640 1 --2- 172.21.15.27:0/85486453 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fe48c06eb70 0x7fe48c071020 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:19.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.470+0000 7fe4ac400640 1 --2- 172.21.15.27:0/85486453 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4a40714a0 0x7fe4a4119aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:19.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.470+0000 7fe4ac400640 1 --2- 172.21.15.27:0/85486453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe4a406fdb0 0x7fe4a4119560 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:19.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.470+0000 7fe4ac400640 1 -- 172.21.15.27:0/85486453 >> 172.21.15.27:0/85486453 conn(0x7fe4a40f7fb0 msgr2=0x7fe4a40f90f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:19.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.470+0000 7fe4ac400640 1 -- 172.21.15.27:0/85486453 shutdown_connections 2024-04-16T05:56:19.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.470+0000 7fe4ac400640 1 -- 172.21.15.27:0/85486453 wait complete. 2024-04-16T05:56:19.487 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-16T05:56:19.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.714+0000 7f4c1be00640 1 -- 172.21.15.27:0/470057245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c1c102040 msgr2=0x7f4c1c102410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:19.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.714+0000 7f4c1be00640 1 --2- 172.21.15.27:0/470057245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c1c102040 0x7f4c1c102410 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c009c50 tx=0x7f4c0c004d10 comp rx=0 tx=0).stop 2024-04-16T05:56:19.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.715+0000 7f4c1be00640 1 -- 172.21.15.27:0/470057245 shutdown_connections 2024-04-16T05:56:19.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.715+0000 7f4c1be00640 1 --2- 172.21.15.27:0/470057245 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c1c1029e0 0x7f4c1c064400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:19.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.715+0000 7f4c1be00640 1 --2- 172.21.15.27:0/470057245 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c1c102040 0x7f4c1c102410 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:19.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.715+0000 7f4c1be00640 1 -- 172.21.15.27:0/470057245 >> 172.21.15.27:0/470057245 conn(0x7f4c1c071df0 msgr2=0x7f4c1c0721f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:19.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.715+0000 7f4c1be00640 1 -- 172.21.15.27:0/470057245 shutdown_connections 2024-04-16T05:56:19.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.715+0000 7f4c1be00640 1 -- 172.21.15.27:0/470057245 wait complete. 2024-04-16T05:56:19.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.716+0000 7f4c1be00640 1 Processor -- start 2024-04-16T05:56:19.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.716+0000 7f4c1be00640 1 -- start start 2024-04-16T05:56:19.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.716+0000 7f4c1be00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c1c102040 0x7f4c1c114e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:19.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.716+0000 7f4c1be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c1c1029e0 0x7f4c1c115360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:19.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.716+0000 7f4c1be00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c115960 con 0x7f4c1c1029e0 2024-04-16T05:56:19.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.716+0000 7f4c1be00640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1c10f000 con 0x7f4c1c102040 2024-04-16T05:56:19.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.717+0000 7f4c1a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c1c1029e0 0x7f4c1c115360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:19.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.717+0000 7f4c1a000640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c1c1029e0 0x7f4c1c115360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58906/0 (socket says 172.21.15.27:58906) 2024-04-16T05:56:19.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.717+0000 7f4c1a000640 1 -- 172.21.15.27:0/1174939902 learned_addr learned my addr 172.21.15.27:0/1174939902 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:19.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.717+0000 7f4c1a000640 1 -- 172.21.15.27:0/1174939902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c1c102040 msgr2=0x7f4c1c114e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:19.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.717+0000 7f4c1aa00640 1 --2- 172.21.15.27:0/1174939902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c1c102040 0x7f4c1c114e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:19.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.717+0000 7f4c1a000640 1 --2- 172.21.15.27:0/1174939902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c1c102040 0x7f4c1c114e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:19.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.717+0000 7f4c1a000640 1 -- 172.21.15.27:0/1174939902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0c0098b0 con 0x7f4c1c1029e0 2024-04-16T05:56:19.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.718+0000 7f4c1a000640 1 --2- 172.21.15.27:0/1174939902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c1c1029e0 0x7f4c1c115360 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4c04002840 tx=0x7f4c04004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:19.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.718+0000 7f4c12a00640 1 -- 172.21.15.27:0/1174939902 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c04007c30 con 0x7f4c1c1029e0 2024-04-16T05:56:19.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.718+0000 7f4c12a00640 1 -- 172.21.15.27:0/1174939902 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4c0400f040 con 0x7f4c1c1029e0 2024-04-16T05:56:19.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.718+0000 7f4c12a00640 1 -- 172.21.15.27:0/1174939902 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4c04009590 con 0x7f4c1c1029e0 2024-04-16T05:56:19.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.718+0000 7f4c1be00640 1 -- 172.21.15.27:0/1174939902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c1c10f2e0 con 0x7f4c1c1029e0 2024-04-16T05:56:19.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.718+0000 7f4c1be00640 1 -- 172.21.15.27:0/1174939902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c1c10f800 con 0x7f4c1c1029e0 2024-04-16T05:56:19.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.719+0000 7f4c1be00640 1 -- 172.21.15.27:0/1174939902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c1c0fd7c0 con 0x7f4c1c1029e0 2024-04-16T05:56:19.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.720+0000 7f4c12a00640 1 -- 172.21.15.27:0/1174939902 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4c0400d6b0 con 0x7f4c1c1029e0 2024-04-16T05:56:19.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.722+0000 7f4c12a00640 1 --2- 172.21.15.27:0/1174939902 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4c0006eb80 0x7f4c00071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:19.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.722+0000 7f4c12a00640 1 -- 172.21.15.27:0/1174939902 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f4c0408e2c0 con 0x7f4c1c1029e0 2024-04-16T05:56:19.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.722+0000 7f4c1aa00640 1 --2- 172.21.15.27:0/1174939902 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4c0006eb80 0x7f4c00071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:19.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.723+0000 7f4c1aa00640 1 --2- 172.21.15.27:0/1174939902 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4c0006eb80 0x7f4c00071030 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c005010 tx=0x7f4c0c02e040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:19.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.727+0000 7f4c12a00640 1 -- 172.21.15.27:0/1174939902 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4c04056e50 con 0x7f4c1c1029e0 2024-04-16T05:56:19.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.935+0000 7f4c1be00640 1 -- 172.21.15.27:0/1174939902 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f4c1c110490 con 0x7f4c1c1029e0 2024-04-16T05:56:19.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.936+0000 7f4c12a00640 1 -- 172.21.15.27:0/1174939902 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v88) v1 ==== 107+0+11 (secure 0 0 0) 0x7f4c0405a510 con 0x7f4c1c1029e0 2024-04-16T05:56:19.934 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-16T05:56:19.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.939+0000 7f4c1be00640 1 -- 172.21.15.27:0/1174939902 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4c0006eb80 msgr2=0x7f4c00071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:19.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.939+0000 7f4c1be00640 1 --2- 172.21.15.27:0/1174939902 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4c0006eb80 0x7f4c00071030 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c005010 tx=0x7f4c0c02e040 comp rx=0 tx=0).stop 2024-04-16T05:56:19.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.939+0000 7f4c1be00640 1 -- 172.21.15.27:0/1174939902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c1c1029e0 msgr2=0x7f4c1c115360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:19.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.939+0000 7f4c1be00640 1 --2- 172.21.15.27:0/1174939902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c1c1029e0 0x7f4c1c115360 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4c04002840 tx=0x7f4c04004570 comp rx=0 tx=0).stop 2024-04-16T05:56:19.937 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.939+0000 7f4c1be00640 1 -- 172.21.15.27:0/1174939902 shutdown_connections 2024-04-16T05:56:19.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.939+0000 7f4c1be00640 1 --2- 172.21.15.27:0/1174939902 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4c0006eb80 0x7f4c00071030 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:19.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.939+0000 7f4c1be00640 1 --2- 172.21.15.27:0/1174939902 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c1c1029e0 0x7f4c1c115360 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:19.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.939+0000 7f4c1be00640 1 --2- 172.21.15.27:0/1174939902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c1c102040 0x7f4c1c114e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:19.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.939+0000 7f4c1be00640 1 -- 172.21.15.27:0/1174939902 >> 172.21.15.27:0/1174939902 conn(0x7f4c1c071df0 msgr2=0x7f4c1c102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:19.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.940+0000 7f4c1be00640 1 -- 172.21.15.27:0/1174939902 shutdown_connections 2024-04-16T05:56:19.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:19.940+0000 7f4c1be00640 1 -- 172.21.15.27:0/1174939902 wait complete. 2024-04-16T05:56:19.956 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-16T05:56:20.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:19 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/382879530' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T05:56:20.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:19 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/85486453' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T05:56:20.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:19 smithi122 ceph-mon[80930]: pgmap v367: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:20.040 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:19 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/382879530' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T05:56:20.041 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:19 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/85486453' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T05:56:20.041 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:19 smithi027 ceph-mon[93635]: pgmap v367: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:20.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.180+0000 7f7af5c00640 1 -- 172.21.15.27:0/3055942280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af0070100 msgr2=0x7f7af01083e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:20.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.180+0000 7f7af5c00640 1 --2- 172.21.15.27:0/3055942280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af0070100 0x7f7af01083e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7adc0048d0 tx=0x7f7adc009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:20.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.180+0000 7f7af5c00640 1 -- 172.21.15.27:0/3055942280 shutdown_connections 2024-04-16T05:56:20.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.180+0000 7f7af5c00640 1 --2- 172.21.15.27:0/3055942280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af0070100 0x7f7af01083e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:20.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.180+0000 7f7af5c00640 1 --2- 172.21.15.27:0/3055942280 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7af006f7f0 0x7f7af006fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:20.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.180+0000 7f7af5c00640 1 -- 172.21.15.27:0/3055942280 >> 172.21.15.27:0/3055942280 conn(0x7f7af00f7c20 msgr2=0x7f7af00fa030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:20.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.181+0000 7f7af5c00640 1 -- 172.21.15.27:0/3055942280 shutdown_connections 2024-04-16T05:56:20.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.181+0000 7f7af5c00640 1 -- 172.21.15.27:0/3055942280 wait complete. 2024-04-16T05:56:20.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.181+0000 7f7af5c00640 1 Processor -- start 2024-04-16T05:56:20.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.182+0000 7f7af5c00640 1 -- start start 2024-04-16T05:56:20.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.182+0000 7f7af5c00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af006f7f0 0x7f7af0114de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:20.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.182+0000 7f7af5c00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7af0070100 0x7f7af0115320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:20.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.182+0000 7f7af5c00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af0115920 con 0x7f7af006f7f0 2024-04-16T05:56:20.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.182+0000 7f7af5c00640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af010ef70 con 0x7f7af0070100 2024-04-16T05:56:20.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.183+0000 7f7aefe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af006f7f0 0x7f7af0114de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:20.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.183+0000 7f7aefe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af006f7f0 0x7f7af0114de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58920/0 (socket says 172.21.15.27:58920) 2024-04-16T05:56:20.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.183+0000 7f7aefe00640 1 -- 172.21.15.27:0/3339750493 learned_addr learned my addr 172.21.15.27:0/3339750493 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:20.181 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.183+0000 7f7aef400640 1 --2- 172.21.15.27:0/3339750493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7af0070100 0x7f7af0115320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:20.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.183+0000 7f7aefe00640 1 -- 172.21.15.27:0/3339750493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7af0070100 msgr2=0x7f7af0115320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:20.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.183+0000 7f7aefe00640 1 --2- 172.21.15.27:0/3339750493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7af0070100 0x7f7af0115320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:20.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.183+0000 7f7aefe00640 1 -- 172.21.15.27:0/3339750493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7adc005040 con 0x7f7af006f7f0 2024-04-16T05:56:20.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.183+0000 7f7aefe00640 1 --2- 172.21.15.27:0/3339750493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af006f7f0 0x7f7af0114de0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0002a50 tx=0x7f7ae0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:20.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.184+0000 7f7ae7e00640 1 -- 172.21.15.27:0/3339750493 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ae0007c30 con 0x7f7af006f7f0 2024-04-16T05:56:20.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.184+0000 7f7ae7e00640 1 -- 172.21.15.27:0/3339750493 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ae000f040 con 0x7f7af006f7f0 2024-04-16T05:56:20.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.184+0000 7f7ae7e00640 1 -- 172.21.15.27:0/3339750493 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ae000dad0 con 0x7f7af006f7f0 2024-04-16T05:56:20.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.184+0000 7f7af5c00640 1 -- 172.21.15.27:0/3339750493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af010f250 con 0x7f7af006f7f0 2024-04-16T05:56:20.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.184+0000 7f7af5c00640 1 -- 172.21.15.27:0/3339750493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af010f740 con 0x7f7af006f7f0 2024-04-16T05:56:20.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.185+0000 7f7af5c00640 1 -- 172.21.15.27:0/3339750493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af0070f30 con 0x7f7af006f7f0 2024-04-16T05:56:20.184 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.187+0000 7f7ae7e00640 1 -- 172.21.15.27:0/3339750493 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7ae000dc70 con 0x7f7af006f7f0 2024-04-16T05:56:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.188+0000 7f7ae7e00640 1 --2- 172.21.15.27:0/3339750493 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7ad406eb80 0x7f7ad4071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.189+0000 7f7ae7e00640 1 -- 172.21.15.27:0/3339750493 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f7ae008e2e0 con 0x7f7af006f7f0 2024-04-16T05:56:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.189+0000 7f7aef400640 1 --2- 172.21.15.27:0/3339750493 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7ad406eb80 0x7f7ad4071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.190+0000 7f7aef400640 1 --2- 172.21.15.27:0/3339750493 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7ad406eb80 0x7f7ad4071030 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7adc0048a0 tx=0x7f7adc009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:20.191 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.194+0000 7f7ae7e00640 1 -- 172.21.15.27:0/3339750493 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7ae0056e70 con 0x7f7af006f7f0 2024-04-16T05:56:20.399 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.401+0000 7f7af5c00640 1 -- 172.21.15.27:0/3339750493 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f7af0110520 con 0x7f7af006f7f0 2024-04-16T05:56:20.400 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.402+0000 7f7ae7e00640 1 -- 172.21.15.27:0/3339750493 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v88) v1 ==== 107+0+11 (secure 0 0 0) 0x7f7ae005a530 con 0x7f7af006f7f0 2024-04-16T05:56:20.400 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-16T05:56:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.405+0000 7f7af5c00640 1 -- 172.21.15.27:0/3339750493 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7ad406eb80 msgr2=0x7f7ad4071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.405+0000 7f7af5c00640 1 --2- 172.21.15.27:0/3339750493 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7ad406eb80 0x7f7ad4071030 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7adc0048a0 tx=0x7f7adc009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.405+0000 7f7af5c00640 1 -- 172.21.15.27:0/3339750493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af006f7f0 msgr2=0x7f7af0114de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.405+0000 7f7af5c00640 1 --2- 172.21.15.27:0/3339750493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af006f7f0 0x7f7af0114de0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0002a50 tx=0x7f7ae0004600 comp rx=0 tx=0).stop 2024-04-16T05:56:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.405+0000 7f7af5c00640 1 -- 172.21.15.27:0/3339750493 shutdown_connections 2024-04-16T05:56:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.405+0000 7f7af5c00640 1 --2- 172.21.15.27:0/3339750493 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7ad406eb80 0x7f7ad4071030 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.405+0000 7f7af5c00640 1 --2- 172.21.15.27:0/3339750493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7af0070100 0x7f7af0115320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.405+0000 7f7af5c00640 1 --2- 172.21.15.27:0/3339750493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af006f7f0 0x7f7af0114de0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.405+0000 7f7af5c00640 1 -- 172.21.15.27:0/3339750493 >> 172.21.15.27:0/3339750493 conn(0x7f7af00f7c20 msgr2=0x7f7af00f9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:20.403 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.405+0000 7f7af5c00640 1 -- 172.21.15.27:0/3339750493 shutdown_connections 2024-04-16T05:56:20.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.405+0000 7f7af5c00640 1 -- 172.21.15.27:0/3339750493 wait complete. 2024-04-16T05:56:20.424 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T05:56:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.658+0000 7f7821400640 1 -- 172.21.15.27:0/3889680546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f781c1026d0 msgr2=0x7f781c102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.658+0000 7f7821400640 1 --2- 172.21.15.27:0/3889680546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f781c1026d0 0x7f781c102b40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f78080048d0 tx=0x7f7808009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.658+0000 7f7821400640 1 -- 172.21.15.27:0/3889680546 shutdown_connections 2024-04-16T05:56:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.658+0000 7f7821400640 1 --2- 172.21.15.27:0/3889680546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f781c1026d0 0x7f781c102b40 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.658+0000 7f7821400640 1 --2- 172.21.15.27:0/3889680546 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f781c1086a0 0x7f781c108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.658+0000 7f7821400640 1 -- 172.21.15.27:0/3889680546 >> 172.21.15.27:0/3889680546 conn(0x7f781c071e70 msgr2=0x7f781c074280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.658+0000 7f7821400640 1 -- 172.21.15.27:0/3889680546 shutdown_connections 2024-04-16T05:56:20.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.658+0000 7f7821400640 1 -- 172.21.15.27:0/3889680546 wait complete. 2024-04-16T05:56:20.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.659+0000 7f7821400640 1 Processor -- start 2024-04-16T05:56:20.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.659+0000 7f7821400640 1 -- start start 2024-04-16T05:56:20.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.660+0000 7f7821400640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f781c1026d0 0x7f781c1190c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:20.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.660+0000 7f7821400640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f781c1086a0 0x7f781c119600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:20.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.660+0000 7f7821400640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781c119c00 con 0x7f781c1026d0 2024-04-16T05:56:20.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.660+0000 7f7821400640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781c119d70 con 0x7f781c1086a0 2024-04-16T05:56:20.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.660+0000 7f781be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f781c1026d0 0x7f781c1190c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:20.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.660+0000 7f781be00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f781c1026d0 0x7f781c1190c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58928/0 (socket says 172.21.15.27:58928) 2024-04-16T05:56:20.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.660+0000 7f781be00640 1 -- 172.21.15.27:0/1995585261 learned_addr learned my addr 172.21.15.27:0/1995585261 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:20.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.661+0000 7f781b400640 1 --2- 172.21.15.27:0/1995585261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f781c1086a0 0x7f781c119600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:20.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.661+0000 7f781be00640 1 -- 172.21.15.27:0/1995585261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f781c1086a0 msgr2=0x7f781c119600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:20.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.661+0000 7f781be00640 1 --2- 172.21.15.27:0/1995585261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f781c1086a0 0x7f781c119600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:20.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.661+0000 7f781be00640 1 -- 172.21.15.27:0/1995585261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7808005040 con 0x7f781c1026d0 2024-04-16T05:56:20.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.661+0000 7f781be00640 1 --2- 172.21.15.27:0/1995585261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f781c1026d0 0x7f781c1190c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f780c002a50 tx=0x7f780c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:20.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.661+0000 7f7813e00640 1 -- 172.21.15.27:0/1995585261 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f780c007c30 con 0x7f781c1026d0 2024-04-16T05:56:20.659 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.661+0000 7f7813e00640 1 -- 172.21.15.27:0/1995585261 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f780c00f040 con 0x7f781c1026d0 2024-04-16T05:56:20.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.661+0000 7f7813e00640 1 -- 172.21.15.27:0/1995585261 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f780c00dad0 con 0x7f781c1026d0 2024-04-16T05:56:20.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.661+0000 7f7821400640 1 -- 172.21.15.27:0/1995585261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f781c113400 con 0x7f781c1026d0 2024-04-16T05:56:20.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.662+0000 7f7821400640 1 -- 172.21.15.27:0/1995585261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f781c1138f0 con 0x7f781c1026d0 2024-04-16T05:56:20.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.663+0000 7f7821400640 1 -- 172.21.15.27:0/1995585261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f781c103dc0 con 0x7f781c1026d0 2024-04-16T05:56:20.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.664+0000 7f7813e00640 1 -- 172.21.15.27:0/1995585261 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f780c00dc70 con 0x7f781c1026d0 2024-04-16T05:56:20.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.665+0000 7f7813e00640 1 --2- 172.21.15.27:0/1995585261 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f780006eb80 0x7f7800071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:20.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.665+0000 7f7813e00640 1 -- 172.21.15.27:0/1995585261 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f780c08e240 con 0x7f781c1026d0 2024-04-16T05:56:20.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.665+0000 7f781b400640 1 --2- 172.21.15.27:0/1995585261 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f780006eb80 0x7f7800071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:20.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.666+0000 7f781b400640 1 --2- 172.21.15.27:0/1995585261 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f780006eb80 0x7f7800071030 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f78080048a0 tx=0x7f7808009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:20.667 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.670+0000 7f7813e00640 1 -- 172.21.15.27:0/1995585261 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f780c056e70 con 0x7f781c1026d0 2024-04-16T05:56:20.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.890+0000 7f7821400640 1 -- 172.21.15.27:0/1995585261 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f781c113b60 con 0x7f781c1026d0 2024-04-16T05:56:20.889 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.892+0000 7f7813e00640 1 -- 172.21.15.27:0/1995585261 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 45 entries v88) v1 ==== 81+0+2673 (secure 0 0 0) 0x7f780c05a530 con 0x7f781c1026d0 2024-04-16T05:56:20.890 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:6825/194115669 2024-04-17T05:51:15.646390+0000 2024-04-16T05:56:20.890 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:6824/194115669 2024-04-17T05:51:15.646390+0000 2024-04-16T05:56:20.890 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6829/4115272713 2024-04-17T05:50:45.882787+0000 2024-04-16T05:56:20.890 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:6827/1078307975 2024-04-17T05:51:44.010470+0000 2024-04-16T05:56:20.890 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6828/4115272713 2024-04-17T05:50:45.882787+0000 2024-04-16T05:56:20.890 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3779871471 2024-04-17T05:45:28.872665+0000 2024-04-16T05:56:20.890 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3077390652 2024-04-17T05:43:00.762041+0000 2024-04-16T05:56:20.890 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/2779081505 2024-04-17T05:32:17.529651+0000 2024-04-16T05:56:20.890 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/2173136120 2024-04-17T05:31:34.994394+0000 2024-04-16T05:56:20.890 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3170976796 2024-04-17T05:31:34.994394+0000 2024-04-16T05:56:20.890 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/2203377971 2024-04-17T05:43:00.762041+0000 2024-04-16T05:56:20.890 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:6826/1078307975 2024-04-17T05:51:44.010470+0000 2024-04-16T05:56:20.890 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:0/1743593984 2024-04-17T05:44:23.662611+0000 2024-04-16T05:56:20.891 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/2779081505 2024-04-17T05:32:17.529651+0000 2024-04-16T05:56:20.891 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/4258676557 2024-04-17T05:32:17.529651+0000 2024-04-16T05:56:20.891 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:0/575464263 2024-04-17T05:44:23.662611+0000 2024-04-16T05:56:20.891 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2398184523 2024-04-17T05:45:28.872665+0000 2024-04-16T05:56:20.891 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/4061328916 2024-04-17T05:43:00.762041+0000 2024-04-16T05:56:20.891 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/99903288 2024-04-17T05:43:00.762041+0000 2024-04-16T05:56:20.891 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6826/1980301393 2024-04-17T05:40:29.686229+0000 2024-04-16T05:56:20.891 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/2700787222 2024-04-17T05:33:49.465983+0000 2024-04-16T05:56:20.891 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/1090905394 2024-04-17T05:45:28.872665+0000 2024-04-16T05:56:20.891 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:0/2486581297 2024-04-17T05:44:23.662611+0000 2024-04-16T05:56:20.891 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/2700787222 2024-04-17T05:33:49.465983+0000 2024-04-16T05:56:20.892 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6827/1980301393 2024-04-17T05:40:29.686229+0000 2024-04-16T05:56:20.892 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3086153578 2024-04-17T05:32:17.529651+0000 2024-04-16T05:56:20.892 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:6828/871482489 2024-04-17T05:44:23.662611+0000 2024-04-16T05:56:20.892 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1860000032 2024-04-17T05:33:49.465983+0000 2024-04-16T05:56:20.892 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:6829/871482489 2024-04-17T05:44:23.662611+0000 2024-04-16T05:56:20.892 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/602364106 2024-04-17T05:33:49.465983+0000 2024-04-16T05:56:20.892 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:0/915233781 2024-04-17T05:44:23.662611+0000 2024-04-16T05:56:20.892 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3775595668 2024-04-17T05:32:17.529651+0000 2024-04-16T05:56:20.893 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:0/2407794149 2024-04-17T05:44:23.662611+0000 2024-04-16T05:56:20.893 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1372853588 2024-04-17T05:45:28.872665+0000 2024-04-16T05:56:20.893 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2242461930 2024-04-17T05:33:49.465983+0000 2024-04-16T05:56:20.893 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2783469284 2024-04-17T05:45:28.872665+0000 2024-04-16T05:56:20.893 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/2173136120 2024-04-17T05:31:34.994394+0000 2024-04-16T05:56:20.893 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3244097830 2024-04-17T05:45:28.872665+0000 2024-04-16T05:56:20.893 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/2203377971 2024-04-17T05:43:00.762041+0000 2024-04-16T05:56:20.893 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:0/3869652783 2024-04-17T05:44:23.662611+0000 2024-04-16T05:56:20.893 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1691369286 2024-04-17T05:45:28.872665+0000 2024-04-16T05:56:20.893 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1435069943 2024-04-17T05:31:34.994394+0000 2024-04-16T05:56:20.893 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3713909498 2024-04-17T05:31:34.994394+0000 2024-04-16T05:56:20.893 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2938698524 2024-04-17T05:43:00.762041+0000 2024-04-16T05:56:20.893 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/1090905394 2024-04-17T05:45:28.872665+0000 2024-04-16T05:56:20.895 INFO:teuthology.orchestra.run.smithi027.stderr:listed 45 entries 2024-04-16T05:56:20.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.895+0000 7f7821400640 1 -- 172.21.15.27:0/1995585261 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f780006eb80 msgr2=0x7f7800071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:20.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.895+0000 7f7821400640 1 --2- 172.21.15.27:0/1995585261 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f780006eb80 0x7f7800071030 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f78080048a0 tx=0x7f7808009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:20.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.895+0000 7f7821400640 1 -- 172.21.15.27:0/1995585261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f781c1026d0 msgr2=0x7f781c1190c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:20.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.895+0000 7f7821400640 1 --2- 172.21.15.27:0/1995585261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f781c1026d0 0x7f781c1190c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f780c002a50 tx=0x7f780c004600 comp rx=0 tx=0).stop 2024-04-16T05:56:20.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.895+0000 7f7821400640 1 -- 172.21.15.27:0/1995585261 shutdown_connections 2024-04-16T05:56:20.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.895+0000 7f7821400640 1 --2- 172.21.15.27:0/1995585261 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f780006eb80 0x7f7800071030 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:20.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.895+0000 7f7821400640 1 --2- 172.21.15.27:0/1995585261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f781c1086a0 0x7f781c119600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:20.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.895+0000 7f7821400640 1 --2- 172.21.15.27:0/1995585261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f781c1026d0 0x7f781c1190c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:20.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.895+0000 7f7821400640 1 -- 172.21.15.27:0/1995585261 >> 172.21.15.27:0/1995585261 conn(0x7f781c071e70 msgr2=0x7f781c073d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:20.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.895+0000 7f7821400640 1 -- 172.21.15.27:0/1995585261 shutdown_connections 2024-04-16T05:56:20.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:20.895+0000 7f7821400640 1 -- 172.21.15.27:0/1995585261 wait complete. 2024-04-16T05:56:20.912 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-04-16T05:56:20.912 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-16T05:56:20.971 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T05:56:21.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:20 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1174939902' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T05:56:21.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:20 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/3339750493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T05:56:21.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:20 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1174939902' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T05:56:21.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:20 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/3339750493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T05:56:21.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.204+0000 7f7ed1800640 1 -- 172.21.15.27:0/1992703711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ecc101120 msgr2=0x7f7ecc1014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:21.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.204+0000 7f7ed1800640 1 --2- 172.21.15.27:0/1992703711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ecc101120 0x7f7ecc1014f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc0048d0 tx=0x7f7ebc009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:21.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.204+0000 7f7ed1800640 1 -- 172.21.15.27:0/1992703711 shutdown_connections 2024-04-16T05:56:21.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.204+0000 7f7ed1800640 1 --2- 172.21.15.27:0/1992703711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ecc101ac0 0x7f7ecc0fb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:21.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.204+0000 7f7ed1800640 1 --2- 172.21.15.27:0/1992703711 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ecc101120 0x7f7ecc1014f0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:21.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.204+0000 7f7ed1800640 1 -- 172.21.15.27:0/1992703711 >> 172.21.15.27:0/1992703711 conn(0x7f7ecc0f6cf0 msgr2=0x7f7ecc0f9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:21.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.204+0000 7f7ed1800640 1 -- 172.21.15.27:0/1992703711 shutdown_connections 2024-04-16T05:56:21.203 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.204+0000 7f7ed1800640 1 -- 172.21.15.27:0/1992703711 wait complete. 2024-04-16T05:56:21.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.205+0000 7f7ed1800640 1 Processor -- start 2024-04-16T05:56:21.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.205+0000 7f7ed1800640 1 -- start start 2024-04-16T05:56:21.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.206+0000 7f7ed1800640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ecc101120 0x7f7ecc114e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:21.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.206+0000 7f7ed1800640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ecc101ac0 0x7f7ecc1153c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:21.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.206+0000 7f7ed1800640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc115a50 con 0x7f7ecc101ac0 2024-04-16T05:56:21.204 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.206+0000 7f7ed1800640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc10f010 con 0x7f7ecc101120 2024-04-16T05:56:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.206+0000 7f7ec3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ecc101ac0 0x7f7ecc1153c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.206+0000 7f7ec3e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ecc101ac0 0x7f7ecc1153c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58952/0 (socket says 172.21.15.27:58952) 2024-04-16T05:56:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.206+0000 7f7ec3e00640 1 -- 172.21.15.27:0/2743914670 learned_addr learned my addr 172.21.15.27:0/2743914670 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.206+0000 7f7ecaa00640 1 --2- 172.21.15.27:0/2743914670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ecc101120 0x7f7ecc114e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.206+0000 7f7ec3e00640 1 -- 172.21.15.27:0/2743914670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ecc101120 msgr2=0x7f7ecc114e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.206+0000 7f7ec3e00640 1 --2- 172.21.15.27:0/2743914670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ecc101120 0x7f7ecc114e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.206+0000 7f7ec3e00640 1 -- 172.21.15.27:0/2743914670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ebc005040 con 0x7f7ecc101ac0 2024-04-16T05:56:21.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.207+0000 7f7ec3e00640 1 --2- 172.21.15.27:0/2743914670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ecc101ac0 0x7f7ecc1153c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7eb400b790 tx=0x7f7eb40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.207+0000 7f7ec2a00640 1 -- 172.21.15.27:0/2743914670 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7eb400c700 con 0x7f7ecc101ac0 2024-04-16T05:56:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.207+0000 7f7ec2a00640 1 -- 172.21.15.27:0/2743914670 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7eb400cd30 con 0x7f7ecc101ac0 2024-04-16T05:56:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.207+0000 7f7ed1800640 1 -- 172.21.15.27:0/2743914670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ecc10f2f0 con 0x7f7ecc101ac0 2024-04-16T05:56:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.207+0000 7f7ec2a00640 1 -- 172.21.15.27:0/2743914670 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7eb4013370 con 0x7f7ecc101ac0 2024-04-16T05:56:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.207+0000 7f7ed1800640 1 -- 172.21.15.27:0/2743914670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ecc10f810 con 0x7f7ecc101ac0 2024-04-16T05:56:21.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.209+0000 7f7ed1800640 1 -- 172.21.15.27:0/2743914670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ecc0fc890 con 0x7f7ecc101ac0 2024-04-16T05:56:21.207 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.209+0000 7f7ec2a00640 1 -- 172.21.15.27:0/2743914670 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f7eb4012070 con 0x7f7ecc101ac0 2024-04-16T05:56:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.210+0000 7f7ec2a00640 1 --2- 172.21.15.27:0/2743914670 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7eac06eb70 0x7f7eac071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.211+0000 7f7ec2a00640 1 -- 172.21.15.27:0/2743914670 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f7eb408e790 con 0x7f7ecc101ac0 2024-04-16T05:56:21.208 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.211+0000 7f7ecaa00640 1 --2- 172.21.15.27:0/2743914670 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7eac06eb70 0x7f7eac071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:21.209 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.211+0000 7f7ecaa00640 1 --2- 172.21.15.27:0/2743914670 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7eac06eb70 0x7f7eac071020 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc004540 tx=0x7f7ebc009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:21.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.215+0000 7f7ec2a00640 1 -- 172.21.15.27:0/2743914670 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f7eb40572a0 con 0x7f7ecc101ac0 2024-04-16T05:56:21.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.422+0000 7f7ed1800640 1 -- 172.21.15.27:0/2743914670 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ecc064380 con 0x7f7ecc101ac0 2024-04-16T05:56:21.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.426+0000 7f7ec2a00640 1 -- 172.21.15.27:0/2743914670 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+17388 (secure 0 0 0) 0x7f7eb405a960 con 0x7f7ecc101ac0 2024-04-16T05:56:21.424 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-04-16T05:56:21.424 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":88,"fsid":"5e3cca06-fbb2-11ee-bc8f-c7b262605968","created":"2024-04-16T05:31:01.142297+0000","modified":"2024-04-16T05:51:44.010556+0000","last_up_change":"2024-04-16T05:50:07.322376+0000","last_in_change":"2024-04-16T05:38:07.718747+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T05:36:52.435427+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-04-16T05:40:10.776546+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-04-16T05:40:11.582658+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4dd7e294-f625-4492-8bd3-18beab34b6b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":56,"up_thru":82,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":1955571017},{"type":"v1","addr":"172.21.15.27:6803","nonce":1955571017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":1955571017},{"type":"v1","addr":"172.21.15.27:6805","nonce":1955571017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":1955571017},{"type":"v1","addr":"172.21.15.27:6809","nonce":1955571017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":1955571017},{"type":"v1","addr":"172.21.15.27:6807","nonce":1955571017}]},"public_addr":"172.21.15.27:6803/1955571017","cluster_addr":"172.21.15.27:6805/1955571017","heartbeat_back_addr":"172.21.15.27:6809/1955571017","heartbeat_front_addr":"172.21.15.27:6807/1955571017","state":["exists","up"]},{"osd":1,"uuid":"abec61aa-7a28-473b-b9d2-04e8d7de8a67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":58,"up_from":62,"up_thru":82,"down_at":59,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":2379867635},{"type":"v1","addr":"172.21.15.27:6811","nonce":2379867635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":2379867635},{"type":"v1","addr":"172.21.15.27:6813","nonce":2379867635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":2379867635},{"type":"v1","addr":"172.21.15.27:6817","nonce":2379867635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":2379867635},{"type":"v1","addr":"172.21.15.27:6815","nonce":2379867635}]},"public_addr":"172.21.15.27:6811/2379867635","cluster_addr":"172.21.15.27:6813/2379867635","heartbeat_back_addr":"172.21.15.27:6817/2379867635","heartbeat_front_addr":"172.21.15.27:6815/2379867635","state":["exists","up"]},{"osd":2,"uuid":"e50778e4-bf1f-45eb-aeab-05e8dad8fa89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":19,"last_clean_end":63,"up_from":67,"up_thru":82,"down_at":64,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":1424147666},{"type":"v1","addr":"172.21.15.27:6819","nonce":1424147666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":1424147666},{"type":"v1","addr":"172.21.15.27:6821","nonce":1424147666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":1424147666},{"type":"v1","addr":"172.21.15.27:6825","nonce":1424147666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":1424147666},{"type":"v1","addr":"172.21.15.27:6823","nonce":1424147666}]},"public_addr":"172.21.15.27:6819/1424147666","cluster_addr":"172.21.15.27:6821/1424147666","heartbeat_back_addr":"172.21.15.27:6825/1424147666","heartbeat_front_addr":"172.21.15.27:6823/1424147666","state":["exists","up"]},{"osd":3,"uuid":"8f55dd70-a9d7-4bc1-a79c-028e66571137","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":68,"up_from":72,"up_thru":82,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3174341439},{"type":"v1","addr":"172.21.15.122:6801","nonce":3174341439}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3174341439},{"type":"v1","addr":"172.21.15.122:6803","nonce":3174341439}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3174341439},{"type":"v1","addr":"172.21.15.122:6807","nonce":3174341439}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3174341439},{"type":"v1","addr":"172.21.15.122:6805","nonce":3174341439}]},"public_addr":"172.21.15.122:6801/3174341439","cluster_addr":"172.21.15.122:6803/3174341439","heartbeat_back_addr":"172.21.15.122:6807/3174341439","heartbeat_front_addr":"172.21.15.122:6805/3174341439","state":["exists","up"]},{"osd":4,"uuid":"ba8734db-cb7a-4242-b354-8ff1f524f25a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":34,"last_clean_end":73,"up_from":77,"up_thru":82,"down_at":74,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2308306674},{"type":"v1","addr":"172.21.15.122:6809","nonce":2308306674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2308306674},{"type":"v1","addr":"172.21.15.122:6811","nonce":2308306674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2308306674},{"type":"v1","addr":"172.21.15.122:6815","nonce":2308306674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2308306674},{"type":"v1","addr":"172.21.15.122:6813","nonce":2308306674}]},"public_addr":"172.21.15.122:6809/2308306674","cluster_addr":"172.21.15.122:6811/2308306674","heartbeat_back_addr":"172.21.15.122:6815/2308306674","heartbeat_front_addr":"172.21.15.122:6813/2308306674","state":["exists","up"]},{"osd":5,"uuid":"6e2753a7-d194-4103-b586-bfef8cbefa2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":39,"last_clean_end":78,"up_from":82,"up_thru":82,"down_at":79,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":3634763653},{"type":"v1","addr":"172.21.15.122:6817","nonce":3634763653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":3634763653},{"type":"v1","addr":"172.21.15.122:6819","nonce":3634763653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":3634763653},{"type":"v1","addr":"172.21.15.122:6823","nonce":3634763653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":3634763653},{"type":"v1","addr":"172.21.15.122:6821","nonce":3634763653}]},"public_addr":"172.21.15.122:6817/3634763653","cluster_addr":"172.21.15.122:6819/3634763653","heartbeat_back_addr":"172.21.15.122:6823/3634763653","heartbeat_front_addr":"172.21.15.122:6821/3634763653","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:35:28.179537+0000","dead_epoch":49},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:36:08.890364+0000","dead_epoch":58},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:36:48.694394+0000","dead_epoch":63},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:37:21.969883+0000","dead_epoch":68},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:37:55.840648+0000","dead_epoch":73},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-16T05:38:29.596986+0000","dead_epoch":78}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":3,"to":2}]},{"pgid":"2.a","mappings":[{"from":3,"to":0}]},{"pgid":"2.15","mappings":[{"from":3,"to":2}]},{"pgid":"3.6","mappings":[{"from":1,"to":5}]},{"pgid":"3.b","mappings":[{"from":1,"to":2}]},{"pgid":"3.d","mappings":[{"from":3,"to":0}]},{"pgid":"3.15","mappings":[{"from":3,"to":2},{"from":4,"to":5}]},{"pgid":"3.1c","mappings":[{"from":4,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.122:6825/194115669":"2024-04-17T05:51:15.646390+0000","172.21.15.122:6824/194115669":"2024-04-17T05:51:15.646390+0000","172.21.15.27:6829/4115272713":"2024-04-17T05:50:45.882787+0000","172.21.15.122:6827/1078307975":"2024-04-17T05:51:44.010470+0000","172.21.15.27:6828/4115272713":"2024-04-17T05:50:45.882787+0000","172.21.15.27:0/3779871471":"2024-04-17T05:45:28.872665+0000","172.21.15.27:0/3077390652":"2024-04-17T05:43:00.762041+0000","172.21.15.27:6800/2779081505":"2024-04-17T05:32:17.529651+0000","172.21.15.27:6801/2173136120":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/3170976796":"2024-04-17T05:31:34.994394+0000","172.21.15.27:6800/2203377971":"2024-04-17T05:43:00.762041+0000","172.21.15.122:6826/1078307975":"2024-04-17T05:51:44.010470+0000","172.21.15.122:0/1743593984":"2024-04-17T05:44:23.662611+0000","172.21.15.27:6801/2779081505":"2024-04-17T05:32:17.529651+0000","172.21.15.27:0/4258676557":"2024-04-17T05:32:17.529651+0000","172.21.15.122:0/575464263":"2024-04-17T05:44:23.662611+0000","172.21.15.27:0/2398184523":"2024-04-17T05:45:28.872665+0000","172.21.15.27:0/4061328916":"2024-04-17T05:43:00.762041+0000","172.21.15.27:0/99903288":"2024-04-17T05:43:00.762041+0000","172.21.15.27:6826/1980301393":"2024-04-17T05:40:29.686229+0000","172.21.15.27:6800/2700787222":"2024-04-17T05:33:49.465983+0000","172.21.15.27:6800/1090905394":"2024-04-17T05:45:28.872665+0000","172.21.15.122:0/2486581297":"2024-04-17T05:44:23.662611+0000","172.21.15.27:6801/2700787222":"2024-04-17T05:33:49.465983+0000","172.21.15.27:6827/1980301393":"2024-04-17T05:40:29.686229+0000","172.21.15.27:0/3086153578":"2024-04-17T05:32:17.529651+0000","172.21.15.122:6828/871482489":"2024-04-17T05:44:23.662611+0000","172.21.15.27:0/1860000032":"2024-04-17T05:33:49.465983+0000","172.21.15.122:6829/871482489":"2024-04-17T05:44:23.662611+0000","172.21.15.27:0/602364106":"2024-04-17T05:33:49.465983+0000","172.21.15.122:0/915233781":"2024-04-17T05:44:23.662611+0000","172.21.15.27:0/3775595668":"2024-04-17T05:32:17.529651+0000","172.21.15.122:0/2407794149":"2024-04-17T05:44:23.662611+0000","172.21.15.27:0/1372853588":"2024-04-17T05:45:28.872665+0000","172.21.15.27:0/2242461930":"2024-04-17T05:33:49.465983+0000","172.21.15.27:0/2783469284":"2024-04-17T05:45:28.872665+0000","172.21.15.27:6800/2173136120":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/3244097830":"2024-04-17T05:45:28.872665+0000","172.21.15.27:6801/2203377971":"2024-04-17T05:43:00.762041+0000","172.21.15.122:0/3869652783":"2024-04-17T05:44:23.662611+0000","172.21.15.27:0/1691369286":"2024-04-17T05:45:28.872665+0000","172.21.15.27:0/1435069943":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/3713909498":"2024-04-17T05:31:34.994394+0000","172.21.15.27:0/2938698524":"2024-04-17T05:43:00.762041+0000","172.21.15.27:6801/1090905394":"2024-04-17T05:45:28.872665+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T05:56:21.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.428+0000 7f7ed1800640 1 -- 172.21.15.27:0/2743914670 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7eac06eb70 msgr2=0x7f7eac071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:21.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.428+0000 7f7ed1800640 1 --2- 172.21.15.27:0/2743914670 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7eac06eb70 0x7f7eac071020 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc004540 tx=0x7f7ebc009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:21.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.428+0000 7f7ed1800640 1 -- 172.21.15.27:0/2743914670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ecc101ac0 msgr2=0x7f7ecc1153c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:21.426 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.428+0000 7f7ed1800640 1 --2- 172.21.15.27:0/2743914670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ecc101ac0 0x7f7ecc1153c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7eb400b790 tx=0x7f7eb40091d0 comp rx=0 tx=0).stop 2024-04-16T05:56:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.428+0000 7f7ed1800640 1 -- 172.21.15.27:0/2743914670 shutdown_connections 2024-04-16T05:56:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.428+0000 7f7ed1800640 1 --2- 172.21.15.27:0/2743914670 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f7eac06eb70 0x7f7eac071020 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.428+0000 7f7ed1800640 1 --2- 172.21.15.27:0/2743914670 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7ecc101ac0 0x7f7ecc1153c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:21.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.428+0000 7f7ed1800640 1 --2- 172.21.15.27:0/2743914670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7ecc101120 0x7f7ecc114e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.429+0000 7f7ed1800640 1 -- 172.21.15.27:0/2743914670 >> 172.21.15.27:0/2743914670 conn(0x7f7ecc0f6cf0 msgr2=0x7f7ecc0fccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.429+0000 7f7ed1800640 1 -- 172.21.15.27:0/2743914670 shutdown_connections 2024-04-16T05:56:21.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.429+0000 7f7ed1800640 1 -- 172.21.15.27:0/2743914670 wait complete. 2024-04-16T05:56:21.448 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-16T05:36:52.435427+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-04-16T05:40:10.776546+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-04-16T05:40:11.582658+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-16T05:56:21.449 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-16T05:56:21.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.681+0000 7fc85dc00640 1 -- 172.21.15.27:0/1094891314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc858104250 msgr2=0x7fc858104620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:21.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.681+0000 7fc85dc00640 1 --2- 172.21.15.27:0/1094891314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc858104250 0x7fc858104620 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc8480048d0 tx=0x7fc848009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:21.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.681+0000 7fc85dc00640 1 -- 172.21.15.27:0/1094891314 shutdown_connections 2024-04-16T05:56:21.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.681+0000 7fc85dc00640 1 --2- 172.21.15.27:0/1094891314 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc858104bf0 0x7fc8580fe280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:21.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.681+0000 7fc85dc00640 1 --2- 172.21.15.27:0/1094891314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc858104250 0x7fc858104620 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:21.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.681+0000 7fc85dc00640 1 -- 172.21.15.27:0/1094891314 >> 172.21.15.27:0/1094891314 conn(0x7fc8580f9fc0 msgr2=0x7fc8580fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:21.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.681+0000 7fc85dc00640 1 -- 172.21.15.27:0/1094891314 shutdown_connections 2024-04-16T05:56:21.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.681+0000 7fc85dc00640 1 -- 172.21.15.27:0/1094891314 wait complete. 2024-04-16T05:56:21.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.682+0000 7fc85dc00640 1 Processor -- start 2024-04-16T05:56:21.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.682+0000 7fc85dc00640 1 -- start start 2024-04-16T05:56:21.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.683+0000 7fc85dc00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc858104250 0x7fc858114d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:21.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.683+0000 7fc85dc00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc858104bf0 0x7fc858115280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:21.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.683+0000 7fc85dc00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc858115880 con 0x7fc858104250 2024-04-16T05:56:21.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.683+0000 7fc85dc00640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85810eed0 con 0x7fc858104bf0 2024-04-16T05:56:21.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.683+0000 7fc857e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc858104250 0x7fc858114d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:21.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.683+0000 7fc857e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc858104250 0x7fc858114d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58978/0 (socket says 172.21.15.27:58978) 2024-04-16T05:56:21.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.683+0000 7fc857e00640 1 -- 172.21.15.27:0/1508884655 learned_addr learned my addr 172.21.15.27:0/1508884655 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:21.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.683+0000 7fc857e00640 1 -- 172.21.15.27:0/1508884655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc858104bf0 msgr2=0x7fc858115280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:21.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.683+0000 7fc857e00640 1 --2- 172.21.15.27:0/1508884655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc858104bf0 0x7fc858115280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:21.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.683+0000 7fc857e00640 1 -- 172.21.15.27:0/1508884655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc848005040 con 0x7fc858104250 2024-04-16T05:56:21.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.684+0000 7fc857e00640 1 --2- 172.21.15.27:0/1508884655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc858104250 0x7fc858114d40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc8480047f0 tx=0x7fc848009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:21.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.684+0000 7fc84fe00640 1 -- 172.21.15.27:0/1508884655 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8480094c0 con 0x7fc858104250 2024-04-16T05:56:21.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.684+0000 7fc84fe00640 1 -- 172.21.15.27:0/1508884655 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc848032e50 con 0x7fc858104250 2024-04-16T05:56:21.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.684+0000 7fc85dc00640 1 -- 172.21.15.27:0/1508884655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc85810f150 con 0x7fc858104250 2024-04-16T05:56:21.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.684+0000 7fc84fe00640 1 -- 172.21.15.27:0/1508884655 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8480433b0 con 0x7fc858104250 2024-04-16T05:56:21.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.684+0000 7fc85dc00640 1 -- 172.21.15.27:0/1508884655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc85810f610 con 0x7fc858104250 2024-04-16T05:56:21.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.685+0000 7fc85dc00640 1 -- 172.21.15.27:0/1508884655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8580ff9c0 con 0x7fc858104250 2024-04-16T05:56:21.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.686+0000 7fc84fe00640 1 -- 172.21.15.27:0/1508884655 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc84802f070 con 0x7fc858104250 2024-04-16T05:56:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.687+0000 7fc84fe00640 1 --2- 172.21.15.27:0/1508884655 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc83c06eb80 0x7fc83c071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.688+0000 7fc84fe00640 1 -- 172.21.15.27:0/1508884655 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7fc8480b0a50 con 0x7fc858104250 2024-04-16T05:56:21.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.688+0000 7fc857400640 1 --2- 172.21.15.27:0/1508884655 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc83c06eb80 0x7fc83c071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:21.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.689+0000 7fc857400640 1 --2- 172.21.15.27:0/1508884655 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc83c06eb80 0x7fc83c071030 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc858110600 tx=0x7fc844009c50 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:21.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.692+0000 7fc84fe00640 1 -- 172.21.15.27:0/1508884655 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc8480795a0 con 0x7fc858104250 2024-04-16T05:56:21.900 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.903+0000 7fc85dc00640 1 -- 172.21.15.27:0/1508884655 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc8581103b0 con 0x7fc858104250 2024-04-16T05:56:21.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.903+0000 7fc84fe00640 1 -- 172.21.15.27:0/1508884655 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc84807cc60 con 0x7fc858104250 2024-04-16T05:56:21.901 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2024-04-16T05:56:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.906+0000 7fc85dc00640 1 -- 172.21.15.27:0/1508884655 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc83c06eb80 msgr2=0x7fc83c071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.906+0000 7fc85dc00640 1 --2- 172.21.15.27:0/1508884655 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc83c06eb80 0x7fc83c071030 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc858110600 tx=0x7fc844009c50 comp rx=0 tx=0).stop 2024-04-16T05:56:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.906+0000 7fc85dc00640 1 -- 172.21.15.27:0/1508884655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc858104250 msgr2=0x7fc858114d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.906+0000 7fc85dc00640 1 --2- 172.21.15.27:0/1508884655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc858104250 0x7fc858114d40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc8480047f0 tx=0x7fc848009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:21.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.906+0000 7fc85dc00640 1 -- 172.21.15.27:0/1508884655 shutdown_connections 2024-04-16T05:56:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.906+0000 7fc85dc00640 1 --2- 172.21.15.27:0/1508884655 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc83c06eb80 0x7fc83c071030 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.906+0000 7fc85dc00640 1 --2- 172.21.15.27:0/1508884655 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc858104bf0 0x7fc858115280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.906+0000 7fc85dc00640 1 --2- 172.21.15.27:0/1508884655 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc858104250 0x7fc858114d40 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.906+0000 7fc85dc00640 1 -- 172.21.15.27:0/1508884655 >> 172.21.15.27:0/1508884655 conn(0x7fc8580f9fc0 msgr2=0x7fc8580fbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.906+0000 7fc85dc00640 1 -- 172.21.15.27:0/1508884655 shutdown_connections 2024-04-16T05:56:21.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:21.907+0000 7fc85dc00640 1 -- 172.21.15.27:0/1508884655 wait complete. 2024-04-16T05:56:21.924 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-04-16T05:56:22.020 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:21 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1995585261' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-16T05:56:22.020 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:21 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/2743914670' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T05:56:22.021 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:21 smithi027 ceph-mon[93635]: pgmap v368: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:22.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:21 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1995585261' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-16T05:56:22.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:21 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/2743914670' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-16T05:56:22.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:21 smithi122 ceph-mon[80930]: pgmap v368: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:22.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.158+0000 7f6a56600640 1 -- 172.21.15.27:0/3438254455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a501029d0 msgr2=0x7f6a500fa130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:22.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.158+0000 7f6a56600640 1 --2- 172.21.15.27:0/3438254455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a501029d0 0x7f6a500fa130 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c0048d0 tx=0x7f6a3c009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:22.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.158+0000 7f6a56600640 1 -- 172.21.15.27:0/3438254455 shutdown_connections 2024-04-16T05:56:22.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.158+0000 7f6a56600640 1 --2- 172.21.15.27:0/3438254455 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a501029d0 0x7f6a500fa130 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:22.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.158+0000 7f6a56600640 1 --2- 172.21.15.27:0/3438254455 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a50102030 0x7f6a50102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:22.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.158+0000 7f6a56600640 1 -- 172.21.15.27:0/3438254455 >> 172.21.15.27:0/3438254455 conn(0x7f6a500f5de0 msgr2=0x7f6a500f81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:22.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.158+0000 7f6a56600640 1 -- 172.21.15.27:0/3438254455 shutdown_connections 2024-04-16T05:56:22.156 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.158+0000 7f6a56600640 1 -- 172.21.15.27:0/3438254455 wait complete. 2024-04-16T05:56:22.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.159+0000 7f6a56600640 1 Processor -- start 2024-04-16T05:56:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.159+0000 7f6a56600640 1 -- start start 2024-04-16T05:56:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.159+0000 7f6a56600640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a50102030 0x7f6a50114d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.159+0000 7f6a56600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a501029d0 0x7f6a501152c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.159+0000 7f6a56600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a501158c0 con 0x7f6a501029d0 2024-04-16T05:56:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.159+0000 7f6a56600640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5010ef10 con 0x7f6a50102030 2024-04-16T05:56:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.160+0000 7f6a4fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a501029d0 0x7f6a501152c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.160+0000 7f6a4fe00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a501029d0 0x7f6a501152c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58986/0 (socket says 172.21.15.27:58986) 2024-04-16T05:56:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.160+0000 7f6a4fe00640 1 -- 172.21.15.27:0/4133269391 learned_addr learned my addr 172.21.15.27:0/4133269391 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.160+0000 7f6a55200640 1 --2- 172.21.15.27:0/4133269391 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a50102030 0x7f6a50114d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.160+0000 7f6a4fe00640 1 -- 172.21.15.27:0/4133269391 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a50102030 msgr2=0x7f6a50114d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.160+0000 7f6a4fe00640 1 --2- 172.21.15.27:0/4133269391 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a50102030 0x7f6a50114d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.160+0000 7f6a4fe00640 1 -- 172.21.15.27:0/4133269391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3c005040 con 0x7f6a501029d0 2024-04-16T05:56:22.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.161+0000 7f6a4fe00640 1 --2- 172.21.15.27:0/4133269391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a501029d0 0x7f6a501152c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c02be70 tx=0x7f6a3c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:22.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.161+0000 7f6a4cc00640 1 -- 172.21.15.27:0/4133269391 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a3c0094c0 con 0x7f6a501029d0 2024-04-16T05:56:22.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.161+0000 7f6a4cc00640 1 -- 172.21.15.27:0/4133269391 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a3c009620 con 0x7f6a501029d0 2024-04-16T05:56:22.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.161+0000 7f6a56600640 1 -- 172.21.15.27:0/4133269391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a5010f190 con 0x7f6a501029d0 2024-04-16T05:56:22.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.161+0000 7f6a4cc00640 1 -- 172.21.15.27:0/4133269391 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a3c0435f0 con 0x7f6a501029d0 2024-04-16T05:56:22.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.161+0000 7f6a56600640 1 -- 172.21.15.27:0/4133269391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a5010f650 con 0x7f6a501029d0 2024-04-16T05:56:22.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.162+0000 7f6a56600640 1 -- 172.21.15.27:0/4133269391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a500fb570 con 0x7f6a501029d0 2024-04-16T05:56:22.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.163+0000 7f6a4cc00640 1 -- 172.21.15.27:0/4133269391 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f6a3c02f070 con 0x7f6a501029d0 2024-04-16T05:56:22.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.164+0000 7f6a4cc00640 1 --2- 172.21.15.27:0/4133269391 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6a3806eb80 0x7f6a38071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:22.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.164+0000 7f6a4cc00640 1 -- 172.21.15.27:0/4133269391 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f6a3c0b0880 con 0x7f6a501029d0 2024-04-16T05:56:22.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.165+0000 7f6a55200640 1 --2- 172.21.15.27:0/4133269391 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6a3806eb80 0x7f6a38071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:22.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.165+0000 7f6a55200640 1 --2- 172.21.15.27:0/4133269391 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6a3806eb80 0x7f6a38071030 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6a44004bc0 tx=0x7f6a44009040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:22.167 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.169+0000 7f6a4cc00640 1 -- 172.21.15.27:0/4133269391 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f6a3c079430 con 0x7f6a501029d0 2024-04-16T05:56:22.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.378+0000 7f6a56600640 1 -- 172.21.15.27:0/4133269391 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f6a50109170 con 0x7f6a501029d0 2024-04-16T05:56:22.376 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.379+0000 7f6a4cc00640 1 -- 172.21.15.27:0/4133269391 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v88) v1 ==== 107+0+11 (secure 0 0 0) 0x7f6a3c07caf0 con 0x7f6a501029d0 2024-04-16T05:56:22.376 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-16T05:56:22.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.381+0000 7f6a56600640 1 -- 172.21.15.27:0/4133269391 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6a3806eb80 msgr2=0x7f6a38071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:22.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.381+0000 7f6a56600640 1 --2- 172.21.15.27:0/4133269391 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6a3806eb80 0x7f6a38071030 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6a44004bc0 tx=0x7f6a44009040 comp rx=0 tx=0).stop 2024-04-16T05:56:22.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.381+0000 7f6a56600640 1 -- 172.21.15.27:0/4133269391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a501029d0 msgr2=0x7f6a501152c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:22.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.381+0000 7f6a56600640 1 --2- 172.21.15.27:0/4133269391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a501029d0 0x7f6a501152c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c02be70 tx=0x7f6a3c009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:22.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.382+0000 7f6a56600640 1 -- 172.21.15.27:0/4133269391 shutdown_connections 2024-04-16T05:56:22.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.382+0000 7f6a56600640 1 --2- 172.21.15.27:0/4133269391 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f6a3806eb80 0x7f6a38071030 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:22.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.382+0000 7f6a56600640 1 --2- 172.21.15.27:0/4133269391 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6a501029d0 0x7f6a501152c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:22.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.382+0000 7f6a56600640 1 --2- 172.21.15.27:0/4133269391 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a50102030 0x7f6a50114d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:22.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.382+0000 7f6a56600640 1 -- 172.21.15.27:0/4133269391 >> 172.21.15.27:0/4133269391 conn(0x7f6a500f5de0 msgr2=0x7f6a500fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:22.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.382+0000 7f6a56600640 1 -- 172.21.15.27:0/4133269391 shutdown_connections 2024-04-16T05:56:22.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.382+0000 7f6a56600640 1 -- 172.21.15.27:0/4133269391 wait complete. 2024-04-16T05:56:22.399 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-04-16T05:56:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.636+0000 7fc2aaa00640 1 -- 172.21.15.27:0/1858921934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2a4104210 msgr2=0x7fc2a41045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.636+0000 7fc2aaa00640 1 --2- 172.21.15.27:0/1858921934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2a4104210 0x7fc2a41045e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc2980048d0 tx=0x7fc298009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.636+0000 7fc2aaa00640 1 -- 172.21.15.27:0/1858921934 shutdown_connections 2024-04-16T05:56:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.636+0000 7fc2aaa00640 1 --2- 172.21.15.27:0/1858921934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2a4104bb0 0x7fc2a40fe240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.636+0000 7fc2aaa00640 1 --2- 172.21.15.27:0/1858921934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2a4104210 0x7fc2a41045e0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.636+0000 7fc2aaa00640 1 -- 172.21.15.27:0/1858921934 >> 172.21.15.27:0/1858921934 conn(0x7fc2a40f9fc0 msgr2=0x7fc2a40fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.636+0000 7fc2aaa00640 1 -- 172.21.15.27:0/1858921934 shutdown_connections 2024-04-16T05:56:22.634 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.636+0000 7fc2aaa00640 1 -- 172.21.15.27:0/1858921934 wait complete. 2024-04-16T05:56:22.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.637+0000 7fc2aaa00640 1 Processor -- start 2024-04-16T05:56:22.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.637+0000 7fc2aaa00640 1 -- start start 2024-04-16T05:56:22.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.637+0000 7fc2aaa00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2a4104210 0x7fc2a4114ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:22.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.637+0000 7fc2aaa00640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2a4104bb0 0x7fc2a4115430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:22.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.637+0000 7fc2aaa00640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a4115a30 con 0x7fc2a4104210 2024-04-16T05:56:22.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.637+0000 7fc2aaa00640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a410ee70 con 0x7fc2a4104bb0 2024-04-16T05:56:22.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.637+0000 7fc2a9600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2a4104210 0x7fc2a4114ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:22.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.638+0000 7fc2a9600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2a4104210 0x7fc2a4114ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59014/0 (socket says 172.21.15.27:59014) 2024-04-16T05:56:22.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.638+0000 7fc2a9600640 1 -- 172.21.15.27:0/2929755351 learned_addr learned my addr 172.21.15.27:0/2929755351 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:22.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.638+0000 7fc2a8c00640 1 --2- 172.21.15.27:0/2929755351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2a4104bb0 0x7fc2a4115430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:22.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.638+0000 7fc2a9600640 1 -- 172.21.15.27:0/2929755351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2a4104bb0 msgr2=0x7fc2a4115430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:22.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.638+0000 7fc2a9600640 1 --2- 172.21.15.27:0/2929755351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2a4104bb0 0x7fc2a4115430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:22.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.638+0000 7fc2a9600640 1 -- 172.21.15.27:0/2929755351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc298005040 con 0x7fc2a4104210 2024-04-16T05:56:22.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.638+0000 7fc2a8c00640 1 --2- 172.21.15.27:0/2929755351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2a4104bb0 0x7fc2a4115430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-16T05:56:22.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.638+0000 7fc2a9600640 1 --2- 172.21.15.27:0/2929755351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2a4104210 0x7fc2a4114ef0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc298004930 tx=0x7fc298009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:22.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.639+0000 7fc2a1600640 1 -- 172.21.15.27:0/2929755351 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2980094c0 con 0x7fc2a4104210 2024-04-16T05:56:22.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.639+0000 7fc2a1600640 1 -- 172.21.15.27:0/2929755351 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc298032e50 con 0x7fc2a4104210 2024-04-16T05:56:22.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.639+0000 7fc2a1600640 1 -- 172.21.15.27:0/2929755351 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2980433b0 con 0x7fc2a4104210 2024-04-16T05:56:22.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.639+0000 7fc2aaa00640 1 -- 172.21.15.27:0/2929755351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a410f0f0 con 0x7fc2a4104210 2024-04-16T05:56:22.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.639+0000 7fc2aaa00640 1 -- 172.21.15.27:0/2929755351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a410f5b0 con 0x7fc2a4104210 2024-04-16T05:56:22.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.640+0000 7fc2aaa00640 1 -- 172.21.15.27:0/2929755351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a40ff980 con 0x7fc2a4104210 2024-04-16T05:56:22.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.641+0000 7fc2a1600640 1 -- 172.21.15.27:0/2929755351 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7fc29802f070 con 0x7fc2a4104210 2024-04-16T05:56:22.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.642+0000 7fc2a1600640 1 --2- 172.21.15.27:0/2929755351 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc28c06eb70 0x7fc28c071020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:22.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.642+0000 7fc2a1600640 1 -- 172.21.15.27:0/2929755351 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7fc2980b0890 con 0x7fc2a4104210 2024-04-16T05:56:22.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.642+0000 7fc2a8c00640 1 --2- 172.21.15.27:0/2929755351 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc28c06eb70 0x7fc28c071020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:22.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.643+0000 7fc2a8c00640 1 --2- 172.21.15.27:0/2929755351 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc28c06eb70 0x7fc28c071020 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc294009af0 tx=0x7fc29400b040 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:22.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.647+0000 7fc2a1600640 1 -- 172.21.15.27:0/2929755351 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7fc2980793a0 con 0x7fc2a4104210 2024-04-16T05:56:22.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.855+0000 7fc2aaa00640 1 -- 172.21.15.27:0/2929755351 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fc2a410b350 con 0x7fc2a4104210 2024-04-16T05:56:22.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.856+0000 7fc2a1600640 1 -- 172.21.15.27:0/2929755351 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v88) v1 ==== 107+0+11 (secure 0 0 0) 0x7fc29807ca60 con 0x7fc2a4104210 2024-04-16T05:56:22.854 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2024-04-16T05:56:22.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.859+0000 7fc2aaa00640 1 -- 172.21.15.27:0/2929755351 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc28c06eb70 msgr2=0x7fc28c071020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:22.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.859+0000 7fc2aaa00640 1 --2- 172.21.15.27:0/2929755351 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc28c06eb70 0x7fc28c071020 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc294009af0 tx=0x7fc29400b040 comp rx=0 tx=0).stop 2024-04-16T05:56:22.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.859+0000 7fc2aaa00640 1 -- 172.21.15.27:0/2929755351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2a4104210 msgr2=0x7fc2a4114ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:22.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.859+0000 7fc2aaa00640 1 --2- 172.21.15.27:0/2929755351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2a4104210 0x7fc2a4114ef0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc298004930 tx=0x7fc298009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:22.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.859+0000 7fc2aaa00640 1 -- 172.21.15.27:0/2929755351 shutdown_connections 2024-04-16T05:56:22.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.859+0000 7fc2aaa00640 1 --2- 172.21.15.27:0/2929755351 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7fc28c06eb70 0x7fc28c071020 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:22.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.859+0000 7fc2aaa00640 1 --2- 172.21.15.27:0/2929755351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc2a4104bb0 0x7fc2a4115430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:22.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.859+0000 7fc2aaa00640 1 --2- 172.21.15.27:0/2929755351 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc2a4104210 0x7fc2a4114ef0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:22.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.859+0000 7fc2aaa00640 1 -- 172.21.15.27:0/2929755351 >> 172.21.15.27:0/2929755351 conn(0x7fc2a40f9fc0 msgr2=0x7fc2a40fbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:22.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.859+0000 7fc2aaa00640 1 -- 172.21.15.27:0/2929755351 shutdown_connections 2024-04-16T05:56:22.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:22.859+0000 7fc2aaa00640 1 -- 172.21.15.27:0/2929755351 wait complete. 2024-04-16T05:56:22.874 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-16T05:56:23.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:22 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/1508884655' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T05:56:23.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:22 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/4133269391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T05:56:23.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.114+0000 7f4f2d600640 1 -- 172.21.15.27:0/298168061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f28104bf0 msgr2=0x7f4f280fe280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:23.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.114+0000 7f4f2d600640 1 --2- 172.21.15.27:0/298168061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f28104bf0 0x7f4f280fe280 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4f140048d0 tx=0x7f4f14009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:23.112 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.114+0000 7f4f2d600640 1 -- 172.21.15.27:0/298168061 shutdown_connections 2024-04-16T05:56:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.114+0000 7f4f2d600640 1 --2- 172.21.15.27:0/298168061 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f28104bf0 0x7f4f280fe280 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.114+0000 7f4f2d600640 1 --2- 172.21.15.27:0/298168061 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f28104250 0x7f4f28104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.114+0000 7f4f2d600640 1 -- 172.21.15.27:0/298168061 >> 172.21.15.27:0/298168061 conn(0x7f4f280f9fc0 msgr2=0x7f4f280fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.114+0000 7f4f2d600640 1 -- 172.21.15.27:0/298168061 shutdown_connections 2024-04-16T05:56:23.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.114+0000 7f4f2d600640 1 -- 172.21.15.27:0/298168061 wait complete. 2024-04-16T05:56:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.115+0000 7f4f2d600640 1 Processor -- start 2024-04-16T05:56:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.115+0000 7f4f2d600640 1 -- start start 2024-04-16T05:56:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.116+0000 7f4f2d600640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f28104250 0x7f4f28114d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.116+0000 7f4f2d600640 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f28104bf0 0x7f4f281152b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.116+0000 7f4f2d600640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f281158b0 con 0x7f4f28104250 2024-04-16T05:56:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.116+0000 7f4f2d600640 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2810ef00 con 0x7f4f28104bf0 2024-04-16T05:56:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.116+0000 7f4f27e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f28104250 0x7f4f28114d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.116+0000 7f4f27e00640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f28104250 0x7f4f28114d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59026/0 (socket says 172.21.15.27:59026) 2024-04-16T05:56:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.116+0000 7f4f27e00640 1 -- 172.21.15.27:0/620542636 learned_addr learned my addr 172.21.15.27:0/620542636 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-04-16T05:56:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.116+0000 7f4f27400640 1 --2- 172.21.15.27:0/620542636 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f28104bf0 0x7f4f281152b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.116+0000 7f4f27e00640 1 -- 172.21.15.27:0/620542636 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f28104bf0 msgr2=0x7f4f281152b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.116+0000 7f4f27e00640 1 --2- 172.21.15.27:0/620542636 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f28104bf0 0x7f4f281152b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:23.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.116+0000 7f4f27e00640 1 -- 172.21.15.27:0/620542636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f14005040 con 0x7f4f28104250 2024-04-16T05:56:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.117+0000 7f4f27e00640 1 --2- 172.21.15.27:0/620542636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f28104250 0x7f4f28114d70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4f1800c830 tx=0x7f4f18004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.117+0000 7f4f1fe00640 1 -- 172.21.15.27:0/620542636 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f1800a7d0 con 0x7f4f28104250 2024-04-16T05:56:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.117+0000 7f4f1fe00640 1 -- 172.21.15.27:0/620542636 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f1800ae00 con 0x7f4f28104250 2024-04-16T05:56:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.117+0000 7f4f2d600640 1 -- 172.21.15.27:0/620542636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f2810f1e0 con 0x7f4f28104250 2024-04-16T05:56:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.117+0000 7f4f1fe00640 1 -- 172.21.15.27:0/620542636 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4f180113a0 con 0x7f4f28104250 2024-04-16T05:56:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.117+0000 7f4f2d600640 1 -- 172.21.15.27:0/620542636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f2810f6d0 con 0x7f4f28104250 2024-04-16T05:56:23.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.118+0000 7f4f2d600640 1 -- 172.21.15.27:0/620542636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f280ff9c0 con 0x7f4f28104250 2024-04-16T05:56:23.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.120+0000 7f4f1fe00640 1 -- 172.21.15.27:0/620542636 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 39) v1 ==== 92745+0+0 (secure 0 0 0) 0x7f4f18012400 con 0x7f4f28104250 2024-04-16T05:56:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.121+0000 7f4f1fe00640 1 --2- 172.21.15.27:0/620542636 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4f0c06eb80 0x7f4f0c071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-16T05:56:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.121+0000 7f4f1fe00640 1 -- 172.21.15.27:0/620542636 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 6994+0+0 (secure 0 0 0) 0x7f4f1808ec30 con 0x7f4f28104250 2024-04-16T05:56:23.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.121+0000 7f4f27400640 1 --2- 172.21.15.27:0/620542636 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4f0c06eb80 0x7f4f0c071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-16T05:56:23.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.122+0000 7f4f27400640 1 --2- 172.21.15.27:0/620542636 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4f0c06eb80 0x7f4f0c071030 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4f140048a0 tx=0x7f4f14009da0 comp rx=0 tx=0).ready entity=mgr.34100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-16T05:56:23.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.125+0000 7f4f1fe00640 1 -- 172.21.15.27:0/620542636 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+188370 (secure 0 0 0) 0x7f4f18057860 con 0x7f4f28104250 2024-04-16T05:56:23.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:22 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/1508884655' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-16T05:56:23.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:22 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/4133269391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-04-16T05:56:23.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.348+0000 7f4f2d600640 1 -- 172.21.15.27:0/620542636 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4f2810b3c0 con 0x7f4f28104250 2024-04-16T05:56:23.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.350+0000 7f4f1fe00640 1 -- 172.21.15.27:0/620542636 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 45 entries v88) v1 ==== 81+0+2673 (secure 0 0 0) 0x7f4f1800a930 con 0x7f4f28104250 2024-04-16T05:56:23.347 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:6825/194115669 2024-04-17T05:51:15.646390+0000 2024-04-16T05:56:23.348 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:6824/194115669 2024-04-17T05:51:15.646390+0000 2024-04-16T05:56:23.348 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6829/4115272713 2024-04-17T05:50:45.882787+0000 2024-04-16T05:56:23.348 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:6827/1078307975 2024-04-17T05:51:44.010470+0000 2024-04-16T05:56:23.348 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6828/4115272713 2024-04-17T05:50:45.882787+0000 2024-04-16T05:56:23.348 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3779871471 2024-04-17T05:45:28.872665+0000 2024-04-16T05:56:23.348 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3077390652 2024-04-17T05:43:00.762041+0000 2024-04-16T05:56:23.348 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/2779081505 2024-04-17T05:32:17.529651+0000 2024-04-16T05:56:23.348 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/2173136120 2024-04-17T05:31:34.994394+0000 2024-04-16T05:56:23.348 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3170976796 2024-04-17T05:31:34.994394+0000 2024-04-16T05:56:23.348 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/2203377971 2024-04-17T05:43:00.762041+0000 2024-04-16T05:56:23.348 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:6826/1078307975 2024-04-17T05:51:44.010470+0000 2024-04-16T05:56:23.348 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:0/1743593984 2024-04-17T05:44:23.662611+0000 2024-04-16T05:56:23.348 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/2779081505 2024-04-17T05:32:17.529651+0000 2024-04-16T05:56:23.348 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/4258676557 2024-04-17T05:32:17.529651+0000 2024-04-16T05:56:23.349 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:0/575464263 2024-04-17T05:44:23.662611+0000 2024-04-16T05:56:23.349 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2398184523 2024-04-17T05:45:28.872665+0000 2024-04-16T05:56:23.349 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/4061328916 2024-04-17T05:43:00.762041+0000 2024-04-16T05:56:23.349 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/99903288 2024-04-17T05:43:00.762041+0000 2024-04-16T05:56:23.349 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6826/1980301393 2024-04-17T05:40:29.686229+0000 2024-04-16T05:56:23.349 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/2700787222 2024-04-17T05:33:49.465983+0000 2024-04-16T05:56:23.349 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/1090905394 2024-04-17T05:45:28.872665+0000 2024-04-16T05:56:23.349 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:0/2486581297 2024-04-17T05:44:23.662611+0000 2024-04-16T05:56:23.349 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/2700787222 2024-04-17T05:33:49.465983+0000 2024-04-16T05:56:23.349 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6827/1980301393 2024-04-17T05:40:29.686229+0000 2024-04-16T05:56:23.349 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3086153578 2024-04-17T05:32:17.529651+0000 2024-04-16T05:56:23.349 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:6828/871482489 2024-04-17T05:44:23.662611+0000 2024-04-16T05:56:23.350 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1860000032 2024-04-17T05:33:49.465983+0000 2024-04-16T05:56:23.350 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:6829/871482489 2024-04-17T05:44:23.662611+0000 2024-04-16T05:56:23.350 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/602364106 2024-04-17T05:33:49.465983+0000 2024-04-16T05:56:23.350 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:0/915233781 2024-04-17T05:44:23.662611+0000 2024-04-16T05:56:23.350 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3775595668 2024-04-17T05:32:17.529651+0000 2024-04-16T05:56:23.350 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:0/2407794149 2024-04-17T05:44:23.662611+0000 2024-04-16T05:56:23.350 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1372853588 2024-04-17T05:45:28.872665+0000 2024-04-16T05:56:23.350 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2242461930 2024-04-17T05:33:49.465983+0000 2024-04-16T05:56:23.350 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2783469284 2024-04-17T05:45:28.872665+0000 2024-04-16T05:56:23.351 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6800/2173136120 2024-04-17T05:31:34.994394+0000 2024-04-16T05:56:23.351 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3244097830 2024-04-17T05:45:28.872665+0000 2024-04-16T05:56:23.351 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/2203377971 2024-04-17T05:43:00.762041+0000 2024-04-16T05:56:23.351 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.122:0/3869652783 2024-04-17T05:44:23.662611+0000 2024-04-16T05:56:23.351 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1691369286 2024-04-17T05:45:28.872665+0000 2024-04-16T05:56:23.351 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/1435069943 2024-04-17T05:31:34.994394+0000 2024-04-16T05:56:23.351 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/3713909498 2024-04-17T05:31:34.994394+0000 2024-04-16T05:56:23.351 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:0/2938698524 2024-04-17T05:43:00.762041+0000 2024-04-16T05:56:23.351 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27:6801/1090905394 2024-04-17T05:45:28.872665+0000 2024-04-16T05:56:23.352 INFO:teuthology.orchestra.run.smithi027.stderr:listed 45 entries 2024-04-16T05:56:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.352+0000 7f4f2d600640 1 -- 172.21.15.27:0/620542636 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4f0c06eb80 msgr2=0x7f4f0c071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.352+0000 7f4f2d600640 1 --2- 172.21.15.27:0/620542636 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4f0c06eb80 0x7f4f0c071030 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4f140048a0 tx=0x7f4f14009da0 comp rx=0 tx=0).stop 2024-04-16T05:56:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.352+0000 7f4f2d600640 1 -- 172.21.15.27:0/620542636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f28104250 msgr2=0x7f4f28114d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-16T05:56:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.352+0000 7f4f2d600640 1 --2- 172.21.15.27:0/620542636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f28104250 0x7f4f28114d70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4f1800c830 tx=0x7f4f18004430 comp rx=0 tx=0).stop 2024-04-16T05:56:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.352+0000 7f4f2d600640 1 -- 172.21.15.27:0/620542636 shutdown_connections 2024-04-16T05:56:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.352+0000 7f4f2d600640 1 --2- 172.21.15.27:0/620542636 >> [v2:172.21.15.27:6800/2004337715,v1:172.21.15.27:6801/2004337715] conn(0x7f4f0c06eb80 0x7f4f0c071030 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.352+0000 7f4f2d600640 1 --2- 172.21.15.27:0/620542636 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4f28104bf0 0x7f4f281152b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.352+0000 7f4f2d600640 1 --2- 172.21.15.27:0/620542636 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f28104250 0x7f4f28114d70 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-16T05:56:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.352+0000 7f4f2d600640 1 -- 172.21.15.27:0/620542636 >> 172.21.15.27:0/620542636 conn(0x7f4f280f9fc0 msgr2=0x7f4f280fbe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-16T05:56:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.353+0000 7f4f2d600640 1 -- 172.21.15.27:0/620542636 shutdown_connections 2024-04-16T05:56:23.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-04-16T05:56:23.353+0000 7f4f2d600640 1 -- 172.21.15.27:0/620542636 wait complete. 2024-04-16T05:56:23.368 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi027.front.sepia.ceph.com... 2024-04-16T05:56:23.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-16T05:56:23.368 DEBUG:teuthology.orchestra.run.smithi027:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-16T05:56:23.465 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T05:56:24.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:23 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/2929755351' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T05:56:24.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:23 smithi027 ceph-mon[93635]: from='client.? 172.21.15.27:0/620542636' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-16T05:56:24.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:23 smithi027 ceph-mon[93635]: pgmap v369: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:24.173 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:23 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/2929755351' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-04-16T05:56:24.173 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:23 smithi122 ceph-mon[80930]: from='client.? 172.21.15.27:0/620542636' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-16T05:56:24.173 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:23 smithi122 ceph-mon[80930]: pgmap v369: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:26.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:26 smithi027 ceph-mon[93635]: pgmap v370: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:27.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:26 smithi122 ceph-mon[80930]: pgmap v370: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:28.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:28 smithi027 ceph-mon[93635]: pgmap v371: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:29.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:28 smithi122 ceph-mon[80930]: pgmap v371: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:30.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:56:30.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:56:31.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:30 smithi122 ceph-mon[80930]: pgmap v372: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:31.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:30 smithi027 ceph-mon[93635]: pgmap v372: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:33.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:32 smithi122 ceph-mon[80930]: pgmap v373: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:33.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:32 smithi027 ceph-mon[93635]: pgmap v373: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:35.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:34 smithi122 ceph-mon[80930]: pgmap v374: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:35.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:34 smithi027 ceph-mon[93635]: pgmap v374: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:37.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:36 smithi122 ceph-mon[80930]: pgmap v375: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:37.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:36 smithi027 ceph-mon[93635]: pgmap v375: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:38.809 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:38 smithi027 ceph-mon[93635]: pgmap v376: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:39.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:38 smithi122 ceph-mon[80930]: pgmap v376: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:41.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:40 smithi122 ceph-mon[80930]: pgmap v377: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:41.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:40 smithi027 ceph-mon[93635]: pgmap v377: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:42.860 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:42 smithi027 ceph-mon[93635]: pgmap v378: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:43.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:42 smithi122 ceph-mon[80930]: pgmap v378: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:45.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:44 smithi122 ceph-mon[80930]: pgmap v379: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:45.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:44 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:56:45.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:44 smithi027 ceph-mon[93635]: pgmap v379: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:45.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:56:47.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:46 smithi122 ceph-mon[80930]: pgmap v380: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:47.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:46 smithi027 ceph-mon[93635]: pgmap v380: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:48.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:48 smithi027 ceph-mon[93635]: pgmap v381: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:49.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:48 smithi122 ceph-mon[80930]: pgmap v381: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:51.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:50 smithi122 ceph-mon[80930]: pgmap v382: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:51.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:50 smithi027 ceph-mon[93635]: pgmap v382: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:53.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:52 smithi122 ceph-mon[80930]: pgmap v383: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:53.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:52 smithi027 ceph-mon[93635]: pgmap v383: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:54.962 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:54 smithi027 ceph-mon[93635]: pgmap v384: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:55.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:54 smithi122 ceph-mon[80930]: pgmap v384: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:57.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:56 smithi122 ceph-mon[80930]: pgmap v385: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:57.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:56 smithi027 ceph-mon[93635]: pgmap v385: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:58.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:58 smithi027 ceph-mon[93635]: pgmap v386: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:56:59.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:58 smithi122 ceph-mon[80930]: pgmap v386: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:00.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:56:59 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:57:00.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:56:59 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:57:00.982 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:00 smithi122 ceph-mon[80930]: pgmap v387: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:01.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:00 smithi027 ceph-mon[93635]: pgmap v387: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:03.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:02 smithi122 ceph-mon[80930]: pgmap v388: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:03.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:02 smithi027 ceph-mon[93635]: pgmap v388: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:05.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:04 smithi122 ceph-mon[80930]: pgmap v389: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:05.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:04 smithi027 ceph-mon[93635]: pgmap v389: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:06.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:05 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:57:06.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:05 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:57:07.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:06 smithi122 ceph-mon[80930]: pgmap v390: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:07.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:06 smithi027 ceph-mon[93635]: pgmap v390: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:08.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:07 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:57:08.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:07 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:57:08.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:07 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:57:08.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:07 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:57:08.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:07 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:57:08.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:07 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:57:09.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:08 smithi122 ceph-mon[80930]: pgmap v391: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:09.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:08 smithi027 ceph-mon[93635]: pgmap v391: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:11.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:10 smithi122 ceph-mon[80930]: pgmap v392: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:11.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:10 smithi027 ceph-mon[93635]: pgmap v392: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:13.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:12 smithi122 ceph-mon[80930]: pgmap v393: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:13.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:12 smithi027 ceph-mon[93635]: pgmap v393: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:14.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:13 smithi122 ceph-mon[80930]: pgmap v394: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:14.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:13 smithi027 ceph-mon[93635]: pgmap v394: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:57:15.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:57:16.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:15 smithi122 ceph-mon[80930]: pgmap v395: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:16.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:15 smithi027 ceph-mon[93635]: pgmap v395: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:18.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:18 smithi027 ceph-mon[93635]: pgmap v396: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:19.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:18 smithi122 ceph-mon[80930]: pgmap v396: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:21.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:20 smithi122 ceph-mon[80930]: pgmap v397: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:21.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:20 smithi027 ceph-mon[93635]: pgmap v397: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:23.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:22 smithi122 ceph-mon[80930]: pgmap v398: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:23.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:22 smithi027 ceph-mon[93635]: pgmap v398: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:25.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:24 smithi122 ceph-mon[80930]: pgmap v399: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:25.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:24 smithi027 ceph-mon[93635]: pgmap v399: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:27.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:26 smithi122 ceph-mon[80930]: pgmap v400: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:27.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:26 smithi027 ceph-mon[93635]: pgmap v400: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:28.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:28 smithi027 ceph-mon[93635]: pgmap v401: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:29.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:28 smithi122 ceph-mon[80930]: pgmap v401: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:30.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:57:30.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:57:31.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:30 smithi122 ceph-mon[80930]: pgmap v402: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:31.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:30 smithi027 ceph-mon[93635]: pgmap v402: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:33.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:32 smithi122 ceph-mon[80930]: pgmap v403: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:33.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:32 smithi027 ceph-mon[93635]: pgmap v403: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:35.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:34 smithi122 ceph-mon[80930]: pgmap v404: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:35.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:34 smithi027 ceph-mon[93635]: pgmap v404: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:37.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:36 smithi122 ceph-mon[80930]: pgmap v405: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:37.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:36 smithi027 ceph-mon[93635]: pgmap v405: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:38.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:38 smithi027 ceph-mon[93635]: pgmap v406: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:39.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:38 smithi122 ceph-mon[80930]: pgmap v406: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:41.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:40 smithi122 ceph-mon[80930]: pgmap v407: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:41.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:40 smithi027 ceph-mon[93635]: pgmap v407: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:43.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:42 smithi122 ceph-mon[80930]: pgmap v408: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:43.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:42 smithi027 ceph-mon[93635]: pgmap v408: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:45.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:44 smithi122 ceph-mon[80930]: pgmap v409: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:45.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:44 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:57:45.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:44 smithi027 ceph-mon[93635]: pgmap v409: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:45.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:57:47.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:46 smithi122 ceph-mon[80930]: pgmap v410: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:47.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:46 smithi027 ceph-mon[93635]: pgmap v410: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:48.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:48 smithi027 ceph-mon[93635]: pgmap v411: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:49.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:48 smithi122 ceph-mon[80930]: pgmap v411: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:51.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:50 smithi122 ceph-mon[80930]: pgmap v412: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:51.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:50 smithi027 ceph-mon[93635]: pgmap v412: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:53.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:52 smithi122 ceph-mon[80930]: pgmap v413: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:53.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:52 smithi027 ceph-mon[93635]: pgmap v413: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:55.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:54 smithi122 ceph-mon[80930]: pgmap v414: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:55.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:54 smithi027 ceph-mon[93635]: pgmap v414: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:57.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:56 smithi122 ceph-mon[80930]: pgmap v415: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:57.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:56 smithi027 ceph-mon[93635]: pgmap v415: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:59.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:58 smithi122 ceph-mon[80930]: pgmap v416: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:57:59.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:58 smithi027 ceph-mon[93635]: pgmap v416: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:00.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:57:59 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:58:00.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:57:59 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:58:01.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:00 smithi122 ceph-mon[80930]: pgmap v417: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:01.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:00 smithi027 ceph-mon[93635]: pgmap v417: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:02.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:01 smithi122 ceph-mon[80930]: pgmap v418: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:02.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:01 smithi027 ceph-mon[93635]: pgmap v418: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:05.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:04 smithi122 ceph-mon[80930]: pgmap v419: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:05.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:04 smithi027 ceph-mon[93635]: pgmap v419: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:07.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:06 smithi122 ceph-mon[80930]: pgmap v420: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:07.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:06 smithi027 ceph-mon[93635]: pgmap v420: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:08.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:07 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:58:08.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:07 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:58:08.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:08 smithi027 ceph-mon[93635]: pgmap v421: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:09.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:08 smithi122 ceph-mon[80930]: pgmap v421: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:10.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:09 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:58:10.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:09 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:58:10.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:09 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:58:10.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:09 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:58:10.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:09 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:58:10.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:09 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:58:11.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:10 smithi122 ceph-mon[80930]: pgmap v422: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:11.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:10 smithi027 ceph-mon[93635]: pgmap v422: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:13.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:12 smithi122 ceph-mon[80930]: pgmap v423: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:13.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:12 smithi027 ceph-mon[93635]: pgmap v423: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:14 smithi122 ceph-mon[80930]: pgmap v424: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:58:15.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:14 smithi027 ceph-mon[93635]: pgmap v424: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:15.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:58:16.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:15 smithi122 ceph-mon[80930]: pgmap v425: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:16.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:15 smithi027 ceph-mon[93635]: pgmap v425: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:18.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:18 smithi027 ceph-mon[93635]: pgmap v426: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:19.023 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:18 smithi122 ceph-mon[80930]: pgmap v426: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:21.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:20 smithi122 ceph-mon[80930]: pgmap v427: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:21.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:20 smithi027 ceph-mon[93635]: pgmap v427: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:23.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:22 smithi122 ceph-mon[80930]: pgmap v428: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:23.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:22 smithi027 ceph-mon[93635]: pgmap v428: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:25.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:24 smithi122 ceph-mon[80930]: pgmap v429: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:25.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:24 smithi027 ceph-mon[93635]: pgmap v429: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:27.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:26 smithi122 ceph-mon[80930]: pgmap v430: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:27.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:26 smithi027 ceph-mon[93635]: pgmap v430: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:28.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:28 smithi027 ceph-mon[93635]: pgmap v431: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:29.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:28 smithi122 ceph-mon[80930]: pgmap v431: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:30.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:58:30.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:58:31.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:30 smithi122 ceph-mon[80930]: pgmap v432: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:31.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:30 smithi027 ceph-mon[93635]: pgmap v432: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:33.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:32 smithi122 ceph-mon[80930]: pgmap v433: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:33.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:32 smithi027 ceph-mon[93635]: pgmap v433: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:35.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:34 smithi122 ceph-mon[80930]: pgmap v434: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:35.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:34 smithi027 ceph-mon[93635]: pgmap v434: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:37.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:36 smithi122 ceph-mon[80930]: pgmap v435: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:37.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:36 smithi027 ceph-mon[93635]: pgmap v435: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:38.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:38 smithi027 ceph-mon[93635]: pgmap v436: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:39.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:38 smithi122 ceph-mon[80930]: pgmap v436: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:41.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:40 smithi122 ceph-mon[80930]: pgmap v437: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:41.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:40 smithi027 ceph-mon[93635]: pgmap v437: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:43.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:42 smithi122 ceph-mon[80930]: pgmap v438: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:43.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:42 smithi027 ceph-mon[93635]: pgmap v438: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:45.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:44 smithi122 ceph-mon[80930]: pgmap v439: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:45.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:44 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:58:45.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:44 smithi027 ceph-mon[93635]: pgmap v439: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:45.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:58:47.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:46 smithi122 ceph-mon[80930]: pgmap v440: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:47.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:46 smithi027 ceph-mon[93635]: pgmap v440: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:49.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:48 smithi122 ceph-mon[80930]: pgmap v441: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:49.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:48 smithi027 ceph-mon[93635]: pgmap v441: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:51.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:50 smithi122 ceph-mon[80930]: pgmap v442: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:51.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:50 smithi027 ceph-mon[93635]: pgmap v442: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:53.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:52 smithi122 ceph-mon[80930]: pgmap v443: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:53.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:52 smithi027 ceph-mon[93635]: pgmap v443: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:55.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:54 smithi122 ceph-mon[80930]: pgmap v444: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:55.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:54 smithi027 ceph-mon[93635]: pgmap v444: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:57.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:56 smithi122 ceph-mon[80930]: pgmap v445: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:57.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:56 smithi027 ceph-mon[93635]: pgmap v445: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:59.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:58 smithi122 ceph-mon[80930]: pgmap v446: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:58:59.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:58 smithi027 ceph-mon[93635]: pgmap v446: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:00.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:58:59 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:59:00.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:58:59 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:59:01.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:00 smithi122 ceph-mon[80930]: pgmap v447: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:01.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:00 smithi027 ceph-mon[93635]: pgmap v447: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:02.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:01 smithi122 ceph-mon[80930]: pgmap v448: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:02.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:01 smithi027 ceph-mon[93635]: pgmap v448: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:05.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:04 smithi122 ceph-mon[80930]: pgmap v449: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:05.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:04 smithi027 ceph-mon[93635]: pgmap v449: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:07.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:06 smithi122 ceph-mon[80930]: pgmap v450: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:07.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:06 smithi027 ceph-mon[93635]: pgmap v450: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:08.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:08 smithi027 ceph-mon[93635]: pgmap v451: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:09.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:08 smithi122 ceph-mon[80930]: pgmap v451: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:10.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:09 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:59:10.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:09 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T05:59:11.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:10 smithi122 ceph-mon[80930]: pgmap v452: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:11.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:10 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:59:11.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:10 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:59:11.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:10 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:59:11.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:10 smithi027 ceph-mon[93635]: pgmap v452: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:11.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:10 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T05:59:11.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:10 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T05:59:11.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:10 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T05:59:13.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:12 smithi122 ceph-mon[80930]: pgmap v453: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:13.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:12 smithi027 ceph-mon[93635]: pgmap v453: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:14 smithi122 ceph-mon[80930]: pgmap v454: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:59:15.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:14 smithi027 ceph-mon[93635]: pgmap v454: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:15.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:59:17.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:16 smithi122 ceph-mon[80930]: pgmap v455: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:17.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:16 smithi027 ceph-mon[93635]: pgmap v455: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:18.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:18 smithi027 ceph-mon[93635]: pgmap v456: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:19.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:18 smithi122 ceph-mon[80930]: pgmap v456: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:21.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:20 smithi122 ceph-mon[80930]: pgmap v457: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:21.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:20 smithi027 ceph-mon[93635]: pgmap v457: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:23.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:22 smithi122 ceph-mon[80930]: pgmap v458: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:23.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:22 smithi027 ceph-mon[93635]: pgmap v458: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:25.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:24 smithi122 ceph-mon[80930]: pgmap v459: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:25.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:24 smithi027 ceph-mon[93635]: pgmap v459: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:27.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:26 smithi122 ceph-mon[80930]: pgmap v460: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:27.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:26 smithi027 ceph-mon[93635]: pgmap v460: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:29.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:28 smithi122 ceph-mon[80930]: pgmap v461: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:29.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:28 smithi027 ceph-mon[93635]: pgmap v461: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:30.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:59:30.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:59:31.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:30 smithi122 ceph-mon[80930]: pgmap v462: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:31.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:30 smithi027 ceph-mon[93635]: pgmap v462: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:33.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:32 smithi122 ceph-mon[80930]: pgmap v463: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:33.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:32 smithi027 ceph-mon[93635]: pgmap v463: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:35.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:34 smithi122 ceph-mon[80930]: pgmap v464: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:35.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:34 smithi027 ceph-mon[93635]: pgmap v464: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:37.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:36 smithi122 ceph-mon[80930]: pgmap v465: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:37.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:36 smithi027 ceph-mon[93635]: pgmap v465: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:39.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:38 smithi122 ceph-mon[80930]: pgmap v466: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:39.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:38 smithi027 ceph-mon[93635]: pgmap v466: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:41.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:40 smithi122 ceph-mon[80930]: pgmap v467: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:41.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:40 smithi027 ceph-mon[93635]: pgmap v467: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:43.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:42 smithi122 ceph-mon[80930]: pgmap v468: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:43.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:42 smithi027 ceph-mon[93635]: pgmap v468: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:44.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:43 smithi122 ceph-mon[80930]: pgmap v469: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:44.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:43 smithi027 ceph-mon[93635]: pgmap v469: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:45.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:44 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:59:45.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T05:59:46.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:45 smithi122 ceph-mon[80930]: pgmap v470: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:46.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:45 smithi027 ceph-mon[93635]: pgmap v470: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:48.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:48 smithi027 ceph-mon[93635]: pgmap v471: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:49.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:48 smithi122 ceph-mon[80930]: pgmap v471: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:51.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:50 smithi122 ceph-mon[80930]: pgmap v472: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:51.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:50 smithi027 ceph-mon[93635]: pgmap v472: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:53.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:52 smithi122 ceph-mon[80930]: pgmap v473: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:53.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:52 smithi027 ceph-mon[93635]: pgmap v473: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:55.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:54 smithi122 ceph-mon[80930]: pgmap v474: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:55.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:54 smithi027 ceph-mon[93635]: pgmap v474: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:57.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:56 smithi122 ceph-mon[80930]: pgmap v475: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:57.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:56 smithi027 ceph-mon[93635]: pgmap v475: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:58.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:58 smithi027 ceph-mon[93635]: pgmap v476: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T05:59:59.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:58 smithi122 ceph-mon[80930]: pgmap v476: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:00.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 05:59:59 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T06:00:00.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 05:59:59 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T06:00:01.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:00 smithi122 ceph-mon[80930]: pgmap v477: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:01.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:00 smithi122 ceph-mon[80930]: overall HEALTH_OK 2024-04-16T06:00:01.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:00 smithi027 ceph-mon[93635]: pgmap v477: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:01.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:00 smithi027 ceph-mon[93635]: overall HEALTH_OK 2024-04-16T06:00:03.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:02 smithi122 ceph-mon[80930]: pgmap v478: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:03.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:02 smithi027 ceph-mon[93635]: pgmap v478: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:05.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:04 smithi122 ceph-mon[80930]: pgmap v479: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:05.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:04 smithi027 ceph-mon[93635]: pgmap v479: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:07.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:06 smithi122 ceph-mon[80930]: pgmap v480: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:07.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:06 smithi027 ceph-mon[93635]: pgmap v480: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:09.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:08 smithi122 ceph-mon[80930]: pgmap v481: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:09.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:08 smithi027 ceph-mon[93635]: pgmap v481: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:11.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:10 smithi122 ceph-mon[80930]: pgmap v482: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:11.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:10 smithi027 ceph-mon[93635]: pgmap v482: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:12.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:11 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T06:00:12.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:11 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T06:00:13.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:12 smithi122 ceph-mon[80930]: pgmap v483: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:13.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:12 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T06:00:13.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:12 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T06:00:13.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:12 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T06:00:13.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:12 smithi027 ceph-mon[93635]: pgmap v483: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:13.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:12 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T06:00:13.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:12 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T06:00:13.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:12 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T06:00:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:14 smithi122 ceph-mon[80930]: pgmap v484: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T06:00:15.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:14 smithi027 ceph-mon[93635]: pgmap v484: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:15.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T06:00:17.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:16 smithi122 ceph-mon[80930]: pgmap v485: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:17.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:16 smithi027 ceph-mon[93635]: pgmap v485: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:19.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:18 smithi122 ceph-mon[80930]: pgmap v486: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:19.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:18 smithi027 ceph-mon[93635]: pgmap v486: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:20.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:19 smithi122 ceph-mon[80930]: pgmap v487: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:20.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:19 smithi027 ceph-mon[93635]: pgmap v487: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:23.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:22 smithi122 ceph-mon[80930]: pgmap v488: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:23.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:22 smithi027 ceph-mon[93635]: pgmap v488: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:24.923 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:24 smithi122 ceph-mon[80930]: pgmap v489: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:25.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:24 smithi027 ceph-mon[93635]: pgmap v489: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:27.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:26 smithi122 ceph-mon[80930]: pgmap v490: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:27.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:26 smithi027 ceph-mon[93635]: pgmap v490: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:28.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:28 smithi027 ceph-mon[93635]: pgmap v491: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:29.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:28 smithi122 ceph-mon[80930]: pgmap v491: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:30.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:29 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T06:00:30.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:29 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T06:00:31.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:30 smithi122 ceph-mon[80930]: pgmap v492: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:31.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:30 smithi027 ceph-mon[93635]: pgmap v492: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:33.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:32 smithi122 ceph-mon[80930]: pgmap v493: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:33.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:32 smithi027 ceph-mon[93635]: pgmap v493: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:35.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:34 smithi122 ceph-mon[80930]: pgmap v494: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:35.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:34 smithi027 ceph-mon[93635]: pgmap v494: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:37.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:36 smithi122 ceph-mon[80930]: pgmap v495: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:37.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:36 smithi027 ceph-mon[93635]: pgmap v495: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:39.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:38 smithi122 ceph-mon[80930]: pgmap v496: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:39.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:38 smithi027 ceph-mon[93635]: pgmap v496: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:41.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:40 smithi122 ceph-mon[80930]: pgmap v497: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:41.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:40 smithi027 ceph-mon[93635]: pgmap v497: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:43.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:42 smithi122 ceph-mon[80930]: pgmap v498: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:43.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:42 smithi027 ceph-mon[93635]: pgmap v498: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:45.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:44 smithi122 ceph-mon[80930]: pgmap v499: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:45.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:44 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T06:00:45.058 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:44 smithi027 ceph-mon[93635]: pgmap v499: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:45.059 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:44 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T06:00:47.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:46 smithi122 ceph-mon[80930]: pgmap v500: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:47.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:46 smithi027 ceph-mon[93635]: pgmap v500: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:49.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:48 smithi122 ceph-mon[80930]: pgmap v501: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:49.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:48 smithi027 ceph-mon[93635]: pgmap v501: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:51.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:50 smithi122 ceph-mon[80930]: pgmap v502: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:51.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:50 smithi027 ceph-mon[93635]: pgmap v502: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:53.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:52 smithi122 ceph-mon[80930]: pgmap v503: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:53.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:52 smithi027 ceph-mon[93635]: pgmap v503: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:55.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:54 smithi122 ceph-mon[80930]: pgmap v504: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:55.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:54 smithi027 ceph-mon[93635]: pgmap v504: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:57.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:56 smithi122 ceph-mon[80930]: pgmap v505: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:57.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:56 smithi027 ceph-mon[93635]: pgmap v505: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:59.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:58 smithi122 ceph-mon[80930]: pgmap v506: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:00:59.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:58 smithi027 ceph-mon[93635]: pgmap v506: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:00.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:59 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T06:01:00.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:00:59 smithi027 ceph-mon[93635]: pgmap v507: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:00.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:59 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T06:01:00.271 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:00:59 smithi122 ceph-mon[80930]: pgmap v507: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:03.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:02 smithi122 ceph-mon[80930]: pgmap v508: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:03.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:02 smithi027 ceph-mon[93635]: pgmap v508: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:05.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:04 smithi122 ceph-mon[80930]: pgmap v509: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:05.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:04 smithi027 ceph-mon[93635]: pgmap v509: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:07.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:06 smithi122 ceph-mon[80930]: pgmap v510: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:07.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:06 smithi027 ceph-mon[93635]: pgmap v510: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:08.806 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:08 smithi027 ceph-mon[93635]: pgmap v511: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:09.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:08 smithi122 ceph-mon[80930]: pgmap v511: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:11.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:10 smithi122 ceph-mon[80930]: pgmap v512: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:11.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:10 smithi027 ceph-mon[93635]: pgmap v512: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:13.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:12 smithi122 ceph-mon[80930]: pgmap v513: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:13.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:12 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T06:01:13.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:12 smithi027 ceph-mon[93635]: pgmap v513: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:13.057 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:12 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-16T06:01:14.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:14 smithi027 ceph-mon[93635]: pgmap v514: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:14.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T06:01:14.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T06:01:14.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T06:01:14.807 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:14 smithi027 ceph-mon[93635]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T06:01:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:14 smithi122 ceph-mon[80930]: pgmap v514: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-16T06:01:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-16T06:01:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-16T06:01:15.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:14 smithi122 ceph-mon[80930]: from='mgr.34100 172.21.15.27:0/2709104501' entity='mgr.smithi027.qgzypi' 2024-04-16T06:01:17.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:16 smithi122 ceph-mon[80930]: pgmap v515: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:17.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:16 smithi027 ceph-mon[93635]: pgmap v515: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:19.022 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:18 smithi122 ceph-mon[80930]: pgmap v516: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:19.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:18 smithi027 ceph-mon[93635]: pgmap v516: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:21.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:20 smithi122 ceph-mon[80930]: pgmap v517: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:21.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:20 smithi027 ceph-mon[93635]: pgmap v517: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:22.543 ERROR:tasks.cephfs.fuse_mount:process failed to terminate after unmount. This probably indicates a bug within ceph-fuse. 2024-04-16T06:01:22.544 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-16T06:01:22.547 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-16T06:01:22.564 INFO:tasks.cephadm:Teardown begin 2024-04-16T06:01:22.565 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa/tasks/cephadm.py", line 2236, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-16T06:01:22.566 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-16T06:01:22.689 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-16T06:01:22.794 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-16T06:01:22.794 DEBUG:teuthology.orchestra.run.smithi027:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-16T06:01:22.856 DEBUG:teuthology.orchestra.run.smithi122:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-16T06:01:22.911 INFO:tasks.cephadm:Stopping all daemons... 2024-04-16T06:01:22.911 INFO:tasks.cephadm.mon.smithi027:Stopping mon.smithi027... 2024-04-16T06:01:22.912 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@mon.smithi027 2024-04-16T06:01:23.010 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:22 smithi027 ceph-mon[93635]: pgmap v518: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:23.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:22 smithi122 ceph-mon[80930]: pgmap v518: 65 pgs: 65 active+clean; 584 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-04-16T06:01:23.306 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:23 smithi027 systemd[1]: Stopping Ceph mon.smithi027 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T06:01:23.977 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:23 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027[93631]: 2024-04-16T06:01:23.578+0000 7fdd8a400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi027 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T06:01:23.977 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:23 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027[93631]: 2024-04-16T06:01:23.578+0000 7fdd8a400700 -1 mon.smithi027@0(leader) e3 *** Got Signal Terminated *** 2024-04-16T06:01:23.978 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:23 smithi027 podman[149835]: 2024-04-16 06:01:23.661088224 +0000 UTC m=+0.218439710 container died 299ca337e4ca2f01879e71aeb313a8b644c35532d69e0163715bc02d510b84ba (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0) 2024-04-16T06:01:24.655 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:24 smithi027 podman[149835]: 2024-04-16 06:01:24.354843299 +0000 UTC m=+0.912194846 container cleanup 299ca337e4ca2f01879e71aeb313a8b644c35532d69e0163715bc02d510b84ba (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-16T06:01:24.655 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:24 smithi027 bash[149835]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027 2024-04-16T06:01:25.056 INFO:journalctl@ceph.mon.smithi027.smithi027.stdout:Apr 16 06:01:24 smithi027 podman[149860]: 2024-04-16 06:01:24.656558532 +0000 UTC m=+0.987885974 container remove 299ca337e4ca2f01879e71aeb313a8b644c35532d69e0163715bc02d510b84ba (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi027, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-16T06:01:25.197 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@mon.smithi027.service' 2024-04-16T06:01:25.354 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T06:01:25.354 INFO:tasks.cephadm.mon.smithi027:Stopped mon.smithi027 2024-04-16T06:01:25.354 INFO:tasks.cephadm.mon.smithi122:Stopping mon.smithi122... 2024-04-16T06:01:25.355 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@mon.smithi122 2024-04-16T06:01:25.724 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:25 smithi122 systemd[1]: Stopping Ceph mon.smithi122 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T06:01:26.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:25 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi122[80926]: 2024-04-16T06:01:25.909+0000 7f4c92400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi122 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T06:01:26.021 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:25 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi122[80926]: 2024-04-16T06:01:25.909+0000 7f4c92400700 -1 mon.smithi122@1(peon) e3 *** Got Signal Terminated *** 2024-04-16T06:01:26.443 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:26 smithi122 podman[117324]: 2024-04-16 06:01:26.105488552 +0000 UTC m=+0.327303157 container died 5db5f3037ed0a5942a64aaa1dc8158b261b55fd6ed8e5d301e28c68e845aac41 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi122, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-16T06:01:26.694 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:26 smithi122 podman[117324]: 2024-04-16 06:01:26.652627404 +0000 UTC m=+0.874442029 container cleanup 5db5f3037ed0a5942a64aaa1dc8158b261b55fd6ed8e5d301e28c68e845aac41 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi122, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True) 2024-04-16T06:01:26.694 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:26 smithi122 bash[117324]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi122 2024-04-16T06:01:27.011 INFO:journalctl@ceph.mon.smithi122.smithi122.stdout:Apr 16 06:01:26 smithi122 podman[117338]: 2024-04-16 06:01:26.896327761 +0000 UTC m=+0.770652873 container remove 5db5f3037ed0a5942a64aaa1dc8158b261b55fd6ed8e5d301e28c68e845aac41 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-mon-smithi122, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T06:01:27.333 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@mon.smithi122.service' 2024-04-16T06:01:27.471 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T06:01:27.471 INFO:tasks.cephadm.mon.smithi122:Stopped mon.smithi122 2024-04-16T06:01:27.471 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-16T06:01:27.471 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.0 2024-04-16T06:01:28.057 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 06:01:27 smithi027 systemd[1]: Stopping Ceph osd.0 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T06:01:28.438 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 06:01:28 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0[102321]: 2024-04-16T06:01:28.126+0000 7f327c600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T06:01:28.438 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 06:01:28 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0[102321]: 2024-04-16T06:01:28.126+0000 7f327c600700 -1 osd.0 88 *** Got signal Terminated *** 2024-04-16T06:01:28.438 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 06:01:28 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0[102321]: 2024-04-16T06:01:28.126+0000 7f327c600700 -1 osd.0 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T06:01:33.806 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 06:01:33 smithi027 podman[149967]: 2024-04-16 06:01:33.503478527 +0000 UTC m=+5.543143849 container died cef03b67bc5ab4d7f2b8c94285b9a743e0f1da8591a2972ac73af538827948ff (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-16T06:01:34.306 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 06:01:33 smithi027 podman[149967]: 2024-04-16 06:01:33.993145925 +0000 UTC m=+6.032811284 container cleanup cef03b67bc5ab4d7f2b8c94285b9a743e0f1da8591a2972ac73af538827948ff (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-16T06:01:34.307 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 06:01:33 smithi027 bash[149967]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0 2024-04-16T06:01:34.806 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 06:01:34 smithi027 podman[149985]: 2024-04-16 06:01:34.332227815 +0000 UTC m=+0.811982286 container remove cef03b67bc5ab4d7f2b8c94285b9a743e0f1da8591a2972ac73af538827948ff (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409) 2024-04-16T06:01:35.557 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 06:01:35 smithi027 podman[150075]: 2024-04-16 06:01:35.147486967 +0000 UTC m=+0.044400528 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T06:01:35.557 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 06:01:35 smithi027 podman[150075]: 2024-04-16 06:01:35.278712662 +0000 UTC m=+0.175626258 container create a9a77efb79570a3818d3c40eb59ea416288a7b41701c9a7ca4de0987c8483f7f (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-16T06:01:36.057 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 06:01:35 smithi027 podman[150075]: 2024-04-16 06:01:35.682799636 +0000 UTC m=+0.579713256 container init a9a77efb79570a3818d3c40eb59ea416288a7b41701c9a7ca4de0987c8483f7f (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0) 2024-04-16T06:01:36.057 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 06:01:35 smithi027 podman[150075]: 2024-04-16 06:01:35.696025832 +0000 UTC m=+0.592939462 container start a9a77efb79570a3818d3c40eb59ea416288a7b41701c9a7ca4de0987c8483f7f (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-16T06:01:36.057 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 06:01:35 smithi027 podman[150075]: 2024-04-16 06:01:35.797855077 +0000 UTC m=+0.694768710 container attach a9a77efb79570a3818d3c40eb59ea416288a7b41701c9a7ca4de0987c8483f7f (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T06:01:36.556 INFO:journalctl@ceph.osd.0.smithi027.stdout:Apr 16 06:01:36 smithi027 podman[150075]: 2024-04-16 06:01:36.161156174 +0000 UTC m=+1.058069767 container died a9a77efb79570a3818d3c40eb59ea416288a7b41701c9a7ca4de0987c8483f7f (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-16T06:01:37.082 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.0.service' 2024-04-16T06:01:37.223 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T06:01:37.224 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-16T06:01:37.224 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-16T06:01:37.224 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.1 2024-04-16T06:01:37.806 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 06:01:37 smithi027 systemd[1]: Stopping Ceph osd.1 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T06:01:38.307 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 06:01:37 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1[107971]: 2024-04-16T06:01:37.930+0000 7f8b18000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T06:01:38.307 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 06:01:37 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1[107971]: 2024-04-16T06:01:37.930+0000 7f8b18000700 -1 osd.1 88 *** Got signal Terminated *** 2024-04-16T06:01:38.307 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 06:01:37 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1[107971]: 2024-04-16T06:01:37.930+0000 7f8b18000700 -1 osd.1 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T06:01:43.807 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 06:01:43 smithi027 podman[150181]: 2024-04-16 06:01:43.410151675 +0000 UTC m=+5.640507162 container died 3a6ab1fc6e79d4c1f1c36a51cc12594bf9ac7084b927a2a536cee52c65dfa954 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-16T06:01:44.213 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 06:01:43 smithi027 podman[150181]: 2024-04-16 06:01:43.971289701 +0000 UTC m=+6.201645242 container cleanup 3a6ab1fc6e79d4c1f1c36a51cc12594bf9ac7084b927a2a536cee52c65dfa954 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-16T06:01:44.213 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 06:01:43 smithi027 bash[150181]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1 2024-04-16T06:01:44.556 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 06:01:44 smithi027 podman[150201]: 2024-04-16 06:01:44.214042574 +0000 UTC m=+0.793751927 container remove 3a6ab1fc6e79d4c1f1c36a51cc12594bf9ac7084b927a2a536cee52c65dfa954 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-16T06:01:45.307 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 06:01:45 smithi027 podman[150292]: 2024-04-16 06:01:44.940006238 +0000 UTC m=+0.036948568 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T06:01:45.307 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 06:01:45 smithi027 podman[150292]: 2024-04-16 06:01:45.065041003 +0000 UTC m=+0.161983369 container create 3c04572fdcdce0ea38734d9672d552ed6841a66570fd839c2da422c45abfb4b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-16T06:01:45.806 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 06:01:45 smithi027 podman[150292]: 2024-04-16 06:01:45.435762046 +0000 UTC m=+0.532704453 container init 3c04572fdcdce0ea38734d9672d552ed6841a66570fd839c2da422c45abfb4b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-16T06:01:45.806 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 06:01:45 smithi027 podman[150292]: 2024-04-16 06:01:45.447791913 +0000 UTC m=+0.544734240 container start 3c04572fdcdce0ea38734d9672d552ed6841a66570fd839c2da422c45abfb4b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-16T06:01:45.806 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 06:01:45 smithi027 podman[150292]: 2024-04-16 06:01:45.577079684 +0000 UTC m=+0.674022108 container attach 3c04572fdcdce0ea38734d9672d552ed6841a66570fd839c2da422c45abfb4b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2) 2024-04-16T06:01:46.307 INFO:journalctl@ceph.osd.1.smithi027.stdout:Apr 16 06:01:45 smithi027 podman[150292]: 2024-04-16 06:01:45.917398867 +0000 UTC m=+1.014341191 container died 3c04572fdcdce0ea38734d9672d552ed6841a66570fd839c2da422c45abfb4b9 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T06:01:46.710 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.1.service' 2024-04-16T06:01:46.833 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T06:01:46.833 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-16T06:01:46.833 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-16T06:01:46.833 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.2 2024-04-16T06:01:47.307 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 06:01:47 smithi027 systemd[1]: Stopping Ceph osd.2 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T06:01:47.807 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 06:01:47 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2[113758]: 2024-04-16T06:01:47.506+0000 7f8430c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T06:01:47.807 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 06:01:47 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2[113758]: 2024-04-16T06:01:47.506+0000 7f8430c00700 -1 osd.2 88 *** Got signal Terminated *** 2024-04-16T06:01:47.807 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 06:01:47 smithi027 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2[113758]: 2024-04-16T06:01:47.506+0000 7f8430c00700 -1 osd.2 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T06:01:53.156 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 06:01:52 smithi027 podman[150407]: 2024-04-16 06:01:52.889736242 +0000 UTC m=+5.551764623 container died 3524731d727794ce24e2c35de1c0358ac3a988f5a11249b28485ff02a89e67b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-04-16T06:01:53.549 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 06:01:53 smithi027 podman[150407]: 2024-04-16 06:01:53.274795113 +0000 UTC m=+5.936823504 container cleanup 3524731d727794ce24e2c35de1c0358ac3a988f5a11249b28485ff02a89e67b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-16T06:01:53.549 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 06:01:53 smithi027 bash[150407]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2 2024-04-16T06:01:53.806 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 06:01:53 smithi027 podman[150425]: 2024-04-16 06:01:53.550842109 +0000 UTC m=+0.645752813 container remove 3524731d727794ce24e2c35de1c0358ac3a988f5a11249b28485ff02a89e67b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-16T06:01:54.763 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 06:01:54 smithi027 podman[150514]: 2024-04-16 06:01:54.327611581 +0000 UTC m=+0.044023673 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T06:01:54.763 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 06:01:54 smithi027 podman[150514]: 2024-04-16 06:01:54.476694647 +0000 UTC m=+0.193106762 container create 5dbbf7a54e1d281922c9eaf4f8e6bd32f52af048ad79e139cdad619c64558992 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-16T06:01:54.763 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 06:01:54 smithi027 podman[150514]: 2024-04-16 06:01:54.764897875 +0000 UTC m=+0.481310014 container init 5dbbf7a54e1d281922c9eaf4f8e6bd32f52af048ad79e139cdad619c64558992 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240409) 2024-04-16T06:01:55.057 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 06:01:54 smithi027 podman[150514]: 2024-04-16 06:01:54.776995979 +0000 UTC m=+0.493408044 container start 5dbbf7a54e1d281922c9eaf4f8e6bd32f52af048ad79e139cdad619c64558992 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-16T06:01:55.057 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 06:01:54 smithi027 podman[150514]: 2024-04-16 06:01:54.912512004 +0000 UTC m=+0.628924159 container attach 5dbbf7a54e1d281922c9eaf4f8e6bd32f52af048ad79e139cdad619c64558992 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T06:01:55.521 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 06:01:55 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3[90158]: 2024-04-16T06:01:55.090+0000 7f7339c00700 -1 osd.3 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.994601+0000 front 2024-04-16T06:01:29.994875+0000 (oldest deadline 2024-04-16T06:01:54.694619+0000) 2024-04-16T06:01:55.522 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:01:55 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:01:55.184+0000 7f5590e00700 -1 osd.4 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.846623+0000 front 2024-04-16T06:01:29.846886+0000 (oldest deadline 2024-04-16T06:01:54.546355+0000) 2024-04-16T06:01:55.557 INFO:journalctl@ceph.osd.2.smithi027.stdout:Apr 16 06:01:55 smithi027 podman[150514]: 2024-04-16 06:01:55.25439813 +0000 UTC m=+0.970810246 container died 5dbbf7a54e1d281922c9eaf4f8e6bd32f52af048ad79e139cdad619c64558992 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-16T06:01:55.990 DEBUG:teuthology.orchestra.run.smithi027:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.2.service' 2024-04-16T06:01:56.127 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T06:01:56.127 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-16T06:01:56.127 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-16T06:01:56.127 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.3 2024-04-16T06:01:56.521 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 06:01:56 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3[90158]: 2024-04-16T06:01:56.135+0000 7f7339c00700 -1 osd.3 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.994601+0000 front 2024-04-16T06:01:29.994875+0000 (oldest deadline 2024-04-16T06:01:54.694619+0000) 2024-04-16T06:01:56.522 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 06:01:56 smithi122 systemd[1]: Stopping Ceph osd.3 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T06:01:56.522 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:01:56 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:01:56.135+0000 7f5590e00700 -1 osd.4 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.846623+0000 front 2024-04-16T06:01:29.846886+0000 (oldest deadline 2024-04-16T06:01:54.546355+0000) 2024-04-16T06:01:56.855 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 06:01:56 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3[90158]: 2024-04-16T06:01:56.686+0000 7f733e400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T06:01:56.855 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 06:01:56 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3[90158]: 2024-04-16T06:01:56.686+0000 7f733e400700 -1 osd.3 88 *** Got signal Terminated *** 2024-04-16T06:01:56.855 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 06:01:56 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3[90158]: 2024-04-16T06:01:56.686+0000 7f733e400700 -1 osd.3 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T06:01:57.178 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:01:56 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:01:56.856+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:01:57.178 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:01:57 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:01:57.100+0000 7f5590e00700 -1 osd.4 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.846623+0000 front 2024-04-16T06:01:29.846886+0000 (oldest deadline 2024-04-16T06:01:54.546355+0000) 2024-04-16T06:01:57.521 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 06:01:57 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3[90158]: 2024-04-16T06:01:57.178+0000 7f7339c00700 -1 osd.3 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.994601+0000 front 2024-04-16T06:01:29.994875+0000 (oldest deadline 2024-04-16T06:01:54.694619+0000) 2024-04-16T06:01:58.093 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:01:57 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:01:57.827+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:01:58.093 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:01:58 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:01:58.095+0000 7f5590e00700 -1 osd.4 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.846623+0000 front 2024-04-16T06:01:29.846886+0000 (oldest deadline 2024-04-16T06:01:54.546355+0000) 2024-04-16T06:01:58.521 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 06:01:58 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3[90158]: 2024-04-16T06:01:58.184+0000 7f7339c00700 -1 osd.3 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.994601+0000 front 2024-04-16T06:01:29.994875+0000 (oldest deadline 2024-04-16T06:01:54.694619+0000) 2024-04-16T06:01:59.191 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:01:58 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:01:58.860+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:01:59.191 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:01:59 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:01:59.078+0000 7f5590e00700 -1 osd.4 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.846623+0000 front 2024-04-16T06:01:29.846886+0000 (oldest deadline 2024-04-16T06:01:54.546355+0000) 2024-04-16T06:01:59.521 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 06:01:59 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3[90158]: 2024-04-16T06:01:59.192+0000 7f7339c00700 -1 osd.3 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.994601+0000 front 2024-04-16T06:01:29.994875+0000 (oldest deadline 2024-04-16T06:01:54.694619+0000) 2024-04-16T06:02:00.196 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:00 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:02:00.103+0000 7f5590e00700 -1 osd.4 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.846623+0000 front 2024-04-16T06:01:29.846886+0000 (oldest deadline 2024-04-16T06:01:54.546355+0000) 2024-04-16T06:02:00.196 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:01:59 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:01:59.865+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:02:00.521 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 06:02:00 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3[90158]: 2024-04-16T06:02:00.198+0000 7f7339c00700 -1 osd.3 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.994601+0000 front 2024-04-16T06:01:29.994875+0000 (oldest deadline 2024-04-16T06:01:54.694619+0000) 2024-04-16T06:02:01.142 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:00 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:00.821+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:02:01.521 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:01 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:02:01.144+0000 7f5590e00700 -1 osd.4 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.846623+0000 front 2024-04-16T06:01:29.846886+0000 (oldest deadline 2024-04-16T06:01:54.546355+0000) 2024-04-16T06:02:01.521 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 06:02:01 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3[90158]: 2024-04-16T06:02:01.242+0000 7f7339c00700 -1 osd.3 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.994601+0000 front 2024-04-16T06:01:29.994875+0000 (oldest deadline 2024-04-16T06:01:54.694619+0000) 2024-04-16T06:02:02.115 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:01 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:01.831+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:02:02.455 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 06:02:02 smithi122 podman[117456]: 2024-04-16 06:02:02.147554228 +0000 UTC m=+5.596101142 container died 897d3579f929bfb3dba5fbb1295b605a9c49f88ee2a61b1fde40b675df1d1b8a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-16T06:02:02.455 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:02 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:02:02.117+0000 7f5590e00700 -1 osd.4 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.846623+0000 front 2024-04-16T06:01:29.846886+0000 (oldest deadline 2024-04-16T06:01:54.546355+0000) 2024-04-16T06:02:02.771 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 06:02:02 smithi122 podman[117456]: 2024-04-16 06:02:02.573227878 +0000 UTC m=+6.021774808 container cleanup 897d3579f929bfb3dba5fbb1295b605a9c49f88ee2a61b1fde40b675df1d1b8a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-16T06:02:02.771 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 06:02:02 smithi122 bash[117456]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3 2024-04-16T06:02:03.151 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:02 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:02.803+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:02:03.151 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 06:02:02 smithi122 podman[117485]: 2024-04-16 06:02:02.858154835 +0000 UTC m=+0.702536292 container remove 897d3579f929bfb3dba5fbb1295b605a9c49f88ee2a61b1fde40b675df1d1b8a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T06:02:03.521 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:03 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:02:03.152+0000 7f5590e00700 -1 osd.4 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.846623+0000 front 2024-04-16T06:01:29.846886+0000 (oldest deadline 2024-04-16T06:01:54.546355+0000) 2024-04-16T06:02:04.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:03 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:03.775+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:02:04.021 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 06:02:03 smithi122 podman[117573]: 2024-04-16 06:02:03.632328929 +0000 UTC m=+0.032684018 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T06:02:04.021 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 06:02:03 smithi122 podman[117573]: 2024-04-16 06:02:03.750628603 +0000 UTC m=+0.150983728 container create b406b0fde8c976310e2196c8c3304ce6b8d8d8bc0b175a910dc553cb58d66414 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-deactivate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-16T06:02:04.483 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:04 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:02:04.183+0000 7f5590e00700 -1 osd.4 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.846623+0000 front 2024-04-16T06:01:29.846886+0000 (oldest deadline 2024-04-16T06:01:54.546355+0000) 2024-04-16T06:02:04.483 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 06:02:04 smithi122 podman[117573]: 2024-04-16 06:02:04.029440641 +0000 UTC m=+0.429795773 container init b406b0fde8c976310e2196c8c3304ce6b8d8d8bc0b175a910dc553cb58d66414 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240409) 2024-04-16T06:02:04.483 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 06:02:04 smithi122 podman[117573]: 2024-04-16 06:02:04.042115917 +0000 UTC m=+0.442471034 container start b406b0fde8c976310e2196c8c3304ce6b8d8d8bc0b175a910dc553cb58d66414 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-deactivate, ceph=True, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-16T06:02:04.484 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 06:02:04 smithi122 podman[117573]: 2024-04-16 06:02:04.144478123 +0000 UTC m=+0.544833291 container attach b406b0fde8c976310e2196c8c3304ce6b8d8d8bc0b175a910dc553cb58d66414 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-deactivate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-16T06:02:04.484 INFO:journalctl@ceph.osd.3.smithi122.stdout:Apr 16 06:02:04 smithi122 podman[117573]: 2024-04-16 06:02:04.485986666 +0000 UTC m=+0.886341809 container died b406b0fde8c976310e2196c8c3304ce6b8d8d8bc0b175a910dc553cb58d66414 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-16T06:02:05.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:04 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:04.753+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:02:05.213 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.3.service' 2024-04-16T06:02:05.334 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T06:02:05.335 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-16T06:02:05.335 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-16T06:02:05.335 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.4 2024-04-16T06:02:05.462 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:05 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:02:05.220+0000 7f5590e00700 -1 osd.4 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.846623+0000 front 2024-04-16T06:01:29.846886+0000 (oldest deadline 2024-04-16T06:01:54.546355+0000) 2024-04-16T06:02:05.761 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:05 smithi122 systemd[1]: Stopping Ceph osd.4 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T06:02:06.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:05 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:05.762+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:02:06.021 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:05 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:02:05.956+0000 7f5595600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T06:02:06.021 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:05 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:02:05.956+0000 7f5595600700 -1 osd.4 88 *** Got signal Terminated *** 2024-04-16T06:02:06.021 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:05 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:02:05.956+0000 7f5595600700 -1 osd.4 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T06:02:06.521 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:06 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:02:06.195+0000 7f5590e00700 -1 osd.4 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.846623+0000 front 2024-04-16T06:01:29.846886+0000 (oldest deadline 2024-04-16T06:01:54.546355+0000) 2024-04-16T06:02:07.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:06 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:06.746+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:02:07.521 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:07 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:02:07.221+0000 7f5590e00700 -1 osd.4 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.846623+0000 front 2024-04-16T06:01:29.846886+0000 (oldest deadline 2024-04-16T06:01:54.546355+0000) 2024-04-16T06:02:08.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:07 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:07.741+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:02:08.022 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:07 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:07.741+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-04-16T06:01:41.555466+0000 front 2024-04-16T06:01:41.555639+0000 (oldest deadline 2024-04-16T06:02:06.854919+0000) 2024-04-16T06:02:08.521 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:08 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:02:08.211+0000 7f5590e00700 -1 osd.4 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.846623+0000 front 2024-04-16T06:01:29.846886+0000 (oldest deadline 2024-04-16T06:01:54.546355+0000) 2024-04-16T06:02:08.521 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:08 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:02:08.211+0000 7f5590e00700 -1 osd.4 88 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-04-16T06:01:42.649326+0000 front 2024-04-16T06:01:42.649664+0000 (oldest deadline 2024-04-16T06:02:07.349022+0000) 2024-04-16T06:02:09.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:08 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:08.770+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:02:09.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:08 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:08.770+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-04-16T06:01:41.555466+0000 front 2024-04-16T06:01:41.555639+0000 (oldest deadline 2024-04-16T06:02:06.854919+0000) 2024-04-16T06:02:09.521 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:09 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:02:09.242+0000 7f5590e00700 -1 osd.4 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.846623+0000 front 2024-04-16T06:01:29.846886+0000 (oldest deadline 2024-04-16T06:01:54.546355+0000) 2024-04-16T06:02:09.521 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:09 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:02:09.242+0000 7f5590e00700 -1 osd.4 88 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-04-16T06:01:42.649326+0000 front 2024-04-16T06:01:42.649664+0000 (oldest deadline 2024-04-16T06:02:07.349022+0000) 2024-04-16T06:02:10.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:09 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:09.740+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:02:10.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:09 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:09.740+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-04-16T06:01:41.555466+0000 front 2024-04-16T06:01:41.555639+0000 (oldest deadline 2024-04-16T06:02:06.854919+0000) 2024-04-16T06:02:10.720 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:10 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:02:10.278+0000 7f5590e00700 -1 osd.4 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:29.846623+0000 front 2024-04-16T06:01:29.846886+0000 (oldest deadline 2024-04-16T06:01:54.546355+0000) 2024-04-16T06:02:10.720 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:10 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4[95145]: 2024-04-16T06:02:10.278+0000 7f5590e00700 -1 osd.4 88 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-04-16T06:01:42.649326+0000 front 2024-04-16T06:01:42.649664+0000 (oldest deadline 2024-04-16T06:02:07.349022+0000) 2024-04-16T06:02:11.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:10 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:10.721+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:02:11.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:10 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:10.721+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-04-16T06:01:41.555466+0000 front 2024-04-16T06:01:41.555639+0000 (oldest deadline 2024-04-16T06:02:06.854919+0000) 2024-04-16T06:02:11.756 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:11 smithi122 podman[117678]: 2024-04-16 06:02:11.474667235 +0000 UTC m=+5.654073977 container died fa97758ff7c59ef8fc98c33abe62e6b2993b294271680e5bd274dd053e18642a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-16T06:02:12.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:11 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:11.758+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:02:12.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:11 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:11.758+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-04-16T06:01:41.555466+0000 front 2024-04-16T06:01:41.555639+0000 (oldest deadline 2024-04-16T06:02:06.854919+0000) 2024-04-16T06:02:12.021 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:11 smithi122 podman[117678]: 2024-04-16 06:02:11.900698265 +0000 UTC m=+6.080105009 container cleanup fa97758ff7c59ef8fc98c33abe62e6b2993b294271680e5bd274dd053e18642a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4, GIT_CLEAN=True, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240412.042930-218a691, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-16T06:02:12.021 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:11 smithi122 bash[117678]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4 2024-04-16T06:02:12.521 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:12 smithi122 podman[117699]: 2024-04-16 06:02:12.110166562 +0000 UTC m=+0.620362089 container remove fa97758ff7c59ef8fc98c33abe62e6b2993b294271680e5bd274dd053e18642a (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409) 2024-04-16T06:02:13.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:12 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:12.722+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:02:13.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:12 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:12.722+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-04-16T06:01:41.555466+0000 front 2024-04-16T06:01:41.555639+0000 (oldest deadline 2024-04-16T06:02:06.854919+0000) 2024-04-16T06:02:13.021 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:12 smithi122 podman[117789]: 2024-04-16 06:02:12.766141686 +0000 UTC m=+0.032330734 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T06:02:13.022 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:12 smithi122 podman[117789]: 2024-04-16 06:02:12.877487426 +0000 UTC m=+0.143676506 container create cf6a5854a8206fee9a366bd9f3822465d387de5ed78f9ce4df90addaf8a25df5 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True) 2024-04-16T06:02:13.521 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:13 smithi122 podman[117789]: 2024-04-16 06:02:13.148090513 +0000 UTC m=+0.414279603 container init cf6a5854a8206fee9a366bd9f3822465d387de5ed78f9ce4df90addaf8a25df5 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-16T06:02:13.522 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:13 smithi122 podman[117789]: 2024-04-16 06:02:13.159353856 +0000 UTC m=+0.425542956 container start cf6a5854a8206fee9a366bd9f3822465d387de5ed78f9ce4df90addaf8a25df5 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-deactivate, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-04-16T06:02:13.522 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:13 smithi122 podman[117789]: 2024-04-16 06:02:13.238360708 +0000 UTC m=+0.504549815 container attach cf6a5854a8206fee9a366bd9f3822465d387de5ed78f9ce4df90addaf8a25df5 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T06:02:13.893 INFO:journalctl@ceph.osd.4.smithi122.stdout:Apr 16 06:02:13 smithi122 podman[117789]: 2024-04-16 06:02:13.605553218 +0000 UTC m=+0.871742262 container died cf6a5854a8206fee9a366bd9f3822465d387de5ed78f9ce4df90addaf8a25df5 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-16T06:02:13.893 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:13 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:13.692+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:02:13.893 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:13 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:13.692+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-04-16T06:01:41.555466+0000 front 2024-04-16T06:01:41.555639+0000 (oldest deadline 2024-04-16T06:02:06.854919+0000) 2024-04-16T06:02:13.894 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:13 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:13.692+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-04-16T06:01:51.357906+0000 front 2024-04-16T06:01:51.358247+0000 (oldest deadline 2024-04-16T06:02:13.057437+0000) 2024-04-16T06:02:14.239 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.4.service' 2024-04-16T06:02:14.355 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T06:02:14.356 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-16T06:02:14.356 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-16T06:02:14.356 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.5 2024-04-16T06:02:14.771 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:14 smithi122 systemd[1]: Stopping Ceph osd.5 for 5e3cca06-fbb2-11ee-bc8f-c7b262605968... 2024-04-16T06:02:14.771 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:14 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:14.647+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:02:14.771 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:14 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:14.647+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-04-16T06:01:41.555466+0000 front 2024-04-16T06:01:41.555639+0000 (oldest deadline 2024-04-16T06:02:06.854919+0000) 2024-04-16T06:02:14.771 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:14 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:14.647+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-04-16T06:01:51.357906+0000 front 2024-04-16T06:01:51.358247+0000 (oldest deadline 2024-04-16T06:02:13.057437+0000) 2024-04-16T06:02:15.271 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:14 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:14.976+0000 7f899b400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-16T06:02:15.272 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:14 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:14.976+0000 7f899b400700 -1 osd.5 88 *** Got signal Terminated *** 2024-04-16T06:02:15.272 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:14 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:14.976+0000 7f899b400700 -1 osd.5 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-16T06:02:16.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:15 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:15.687+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:02:16.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:15 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:15.687+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-04-16T06:01:41.555466+0000 front 2024-04-16T06:01:41.555639+0000 (oldest deadline 2024-04-16T06:02:06.854919+0000) 2024-04-16T06:02:16.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:15 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:15.687+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-04-16T06:01:51.357906+0000 front 2024-04-16T06:01:51.358247+0000 (oldest deadline 2024-04-16T06:02:13.057437+0000) 2024-04-16T06:02:17.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:16 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:16.678+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:02:17.022 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:16 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:16.678+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-04-16T06:01:41.555466+0000 front 2024-04-16T06:01:41.555639+0000 (oldest deadline 2024-04-16T06:02:06.854919+0000) 2024-04-16T06:02:17.022 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:16 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:16.678+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-04-16T06:01:51.357906+0000 front 2024-04-16T06:01:51.358247+0000 (oldest deadline 2024-04-16T06:02:13.057437+0000) 2024-04-16T06:02:18.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:17 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:17.691+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:02:18.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:17 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:17.691+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-04-16T06:01:41.555466+0000 front 2024-04-16T06:01:41.555639+0000 (oldest deadline 2024-04-16T06:02:06.854919+0000) 2024-04-16T06:02:18.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:17 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:17.691+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-04-16T06:01:51.357906+0000 front 2024-04-16T06:01:51.358247+0000 (oldest deadline 2024-04-16T06:02:13.057437+0000) 2024-04-16T06:02:19.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:18 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:18.705+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:02:19.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:18 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:18.705+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-04-16T06:01:41.555466+0000 front 2024-04-16T06:01:41.555639+0000 (oldest deadline 2024-04-16T06:02:06.854919+0000) 2024-04-16T06:02:19.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:18 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:18.705+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-04-16T06:01:51.357906+0000 front 2024-04-16T06:01:51.358247+0000 (oldest deadline 2024-04-16T06:02:13.057437+0000) 2024-04-16T06:02:20.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:19 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:19.734+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6806 osd.0 since back 2024-04-16T06:01:32.853266+0000 front 2024-04-16T06:01:32.853634+0000 (oldest deadline 2024-04-16T06:01:56.353115+0000) 2024-04-16T06:02:20.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:19 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:19.734+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6814 osd.1 since back 2024-04-16T06:01:41.555466+0000 front 2024-04-16T06:01:41.555639+0000 (oldest deadline 2024-04-16T06:02:06.854919+0000) 2024-04-16T06:02:20.021 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:19 smithi122 ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5[100421]: 2024-04-16T06:02:19.734+0000 7f8991c00700 -1 osd.5 88 heartbeat_check: no reply from 172.21.15.27:6822 osd.2 since back 2024-04-16T06:01:51.357906+0000 front 2024-04-16T06:01:51.358247+0000 (oldest deadline 2024-04-16T06:02:13.057437+0000) 2024-04-16T06:02:20.771 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:20 smithi122 podman[117892]: 2024-04-16 06:02:20.480117443 +0000 UTC m=+5.628248034 container died fd12260a820d761235c1d30eb3f0b5e36e57d14af4179de7dc0b0ad795477229 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=) 2024-04-16T06:02:21.151 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:20 smithi122 podman[117892]: 2024-04-16 06:02:20.943800833 +0000 UTC m=+6.091931439 container cleanup fd12260a820d761235c1d30eb3f0b5e36e57d14af4179de7dc0b0ad795477229 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240412.042930-218a691) 2024-04-16T06:02:21.151 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:20 smithi122 bash[117892]: ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5 2024-04-16T06:02:21.521 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:21 smithi122 podman[117912]: 2024-04-16 06:02:21.153028679 +0000 UTC m=+0.665872566 container remove fd12260a820d761235c1d30eb3f0b5e36e57d14af4179de7dc0b0ad795477229 (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409) 2024-04-16T06:02:22.198 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:21 smithi122 podman[117998]: 2024-04-16 06:02:21.829080056 +0000 UTC m=+0.034438547 image pull a4e4811815e4b21a2c2eb3d6af4424883cb2ff8e6a7dd48947e0a0b5759e4f05 quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa 2024-04-16T06:02:22.198 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:21 smithi122 podman[117998]: 2024-04-16 06:02:21.928873787 +0000 UTC m=+0.134232270 container create 009673d95cb563e777f49b647fb42948d5c85a8949e04bc2cb43a8074b07450c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-16T06:02:22.198 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:22 smithi122 podman[117998]: 2024-04-16 06:02:22.199921935 +0000 UTC m=+0.405280412 container init 009673d95cb563e777f49b647fb42948d5c85a8949e04bc2cb43a8074b07450c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, CEPH_POINT_RELEASE=) 2024-04-16T06:02:22.521 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:22 smithi122 podman[117998]: 2024-04-16 06:02:22.211886184 +0000 UTC m=+0.417244647 container start 009673d95cb563e777f49b647fb42948d5c85a8949e04bc2cb43a8074b07450c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-vshankar-testing-20240412.042930-218a691, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-16T06:02:22.521 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:22 smithi122 podman[117998]: 2024-04-16 06:02:22.306462947 +0000 UTC m=+0.511821474 container attach 009673d95cb563e777f49b647fb42948d5c85a8949e04bc2cb43a8074b07450c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-16T06:02:22.947 INFO:journalctl@ceph.osd.5.smithi122.stdout:Apr 16 06:02:22 smithi122 podman[117998]: 2024-04-16 06:02:22.655365958 +0000 UTC m=+0.860724448 container died 009673d95cb563e777f49b647fb42948d5c85a8949e04bc2cb43a8074b07450c (image=quay.ceph.io/ceph-ci/ceph@sha256:b2bdb0d9d0471afa314bb0b5ecd318ce61e4dda086bd028e6a35664d56dc32aa, name=ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240412.042930-218a691, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS) 2024-04-16T06:02:23.439 DEBUG:teuthology.orchestra.run.smithi122:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5e3cca06-fbb2-11ee-bc8f-c7b262605968@osd.5.service' 2024-04-16T06:02:23.564 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-16T06:02:23.564 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-16T06:02:23.565 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 --force --keep-logs 2024-04-16T06:02:29.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stdout:ceph_fuse: fuse loop returned - joining 2024-04-16T06:02:29.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stdout:ceph_fuse: fuse loop returned - joined 2024-04-16T06:02:29.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stdout:calling unmount 2024-04-16T06:02:29.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr:ceph-fuse[76806]: fuse finished with error 0 and tester_r 02024-04-16T06:02:29.110+0000 7f6d76acf300 -1 fuse_session_loop_mt 3 -- done 2024-04-16T06:02:29.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stderr: 2024-04-16T06:02:54.124 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 --force --keep-logs 2024-04-16T06:02:59.109 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi027.stdout:calling finalize 2024-04-16T06:03:13.488 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-16T06:03:13.605 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-16T06:03:13.709 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-16T06:03:13.711 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-04-16_05:08:59-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657906/remote/smithi027/crash 2024-04-16T06:03:13.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/crash -- . 2024-04-16T06:03:13.837 INFO:teuthology.orchestra.run.smithi027.stderr:tar: /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/crash: Cannot open: No such file or directory 2024-04-16T06:03:13.837 INFO:teuthology.orchestra.run.smithi027.stderr:tar: Error is not recoverable: exiting now 2024-04-16T06:03:13.844 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/crash to /home/teuthworker/archive/vshankar-2024-04-16_05:08:59-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657906/remote/smithi122/crash 2024-04-16T06:03:13.845 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/crash -- . 2024-04-16T06:03:13.955 INFO:teuthology.orchestra.run.smithi122.stderr:tar: /var/lib/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/crash: Cannot open: No such file or directory 2024-04-16T06:03:13.955 INFO:teuthology.orchestra.run.smithi122.stderr:tar: Error is not recoverable: exiting now 2024-04-16T06:03:13.959 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-16T06:03:13.959 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-16T06:03:14.077 INFO:teuthology.orchestra.run.smithi027.stdout:2024-04-16T05:50:00.000325+0000 mon.smithi027 (mon.0) 516 : cluster [WRN] osd.5 (root=default,host=smithi122) is down 2024-04-16T06:03:14.079 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-16T06:03:14.079 INFO:tasks.cephadm:Compressing logs... 2024-04-16T06:03:14.079 DEBUG:teuthology.orchestra.run.smithi027:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T06:03:14.084 DEBUG:teuthology.orchestra.run.smithi122:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T06:03:14.159 INFO:teuthology.orchestra.run.smithi027.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-16T06:03:14.160 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-16T06:03:14.161 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-04-16T06:03:14.162 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi027.hkhpuq.log 2024-04-16T06:03:14.163 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph.audit.log 2024-04-16T06:03:14.164 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi027.hkhpuq.log: gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-osd.1.log 2024-04-16T06:03:14.166 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-osd.0.log 2024-04-16T06:03:14.168 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mon.smithi027.log 2024-04-16T06:03:14.169 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph.cephadm.log 2024-04-16T06:03:14.172 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mon.smithi027.log: /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph.log 2024-04-16T06:03:14.174 INFO:teuthology.orchestra.run.smithi027.stderr: 91.1% -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph.audit.log.gz 2024-04-16T06:03:14.177 INFO:teuthology.orchestra.run.smithi122.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-16T06:03:14.177 INFO:teuthology.orchestra.run.smithi122.stderr:: No such file or directory 2024-04-16T06:03:14.178 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph.log: 85.3% -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph.cephadm.log.gz 2024-04-16T06:03:14.178 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-osd.2.log 2024-04-16T06:03:14.178 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mgr.smithi122.sgokkq.log 2024-04-16T06:03:14.179 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi122.lojjdp.log 2024-04-16T06:03:14.180 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-volume.log 2024-04-16T06:03:14.180 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mgr.smithi122.sgokkq.log: gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi122.rulewi.log 2024-04-16T06:03:14.181 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi122.lojjdp.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-16T06:03:14.181 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mon.smithi122.log 2024-04-16T06:03:14.182 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mgr.smithi027.qgzypi.log 2024-04-16T06:03:14.183 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi122.rulewi.log: gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph.audit.log 2024-04-16T06:03:14.184 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mon.smithi122.log: gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-osd.4.log 2024-04-16T06:03:14.186 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-osd.5.log 2024-04-16T06:03:14.186 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-osd.2.log: /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-volume.log: 96.6% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-04-16T06:03:14.188 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-osd.3.log 2024-04-16T06:03:14.190 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-osd.5.log: /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph.cephadm.log 2024-04-16T06:03:14.190 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi027.ejoytk.log 2024-04-16T06:03:14.191 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mgr.smithi027.qgzypi.log: 88.6% -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph.log.gz 2024-04-16T06:03:14.193 INFO:teuthology.orchestra.run.smithi122.stderr: 91.3% -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph.audit.log.gz 2024-04-16T06:03:14.194 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph.cephadm.log: 85.2%gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph.log 2024-04-16T06:03:14.196 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-volume.log 2024-04-16T06:03:14.196 INFO:teuthology.orchestra.run.smithi122.stderr: 93.4% -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi122.rulewi.log.gz 2024-04-16T06:03:14.197 INFO:teuthology.orchestra.run.smithi122.stderr: 93.3% -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi122.lojjdp.log.gz 2024-04-16T06:03:14.197 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph.cephadm.log.gz 2024-04-16T06:03:14.198 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.log 2024-04-16T06:03:14.203 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph.log: /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-volume.log: /var/log/ceph/ceph-client.1.log: 88.6% -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph.log.gz 2024-04-16T06:03:14.206 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi027.ejoytk.log: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-16T06:03:14.212 INFO:teuthology.orchestra.run.smithi027.stderr: 90.9% -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-volume.log.gz 2024-04-16T06:03:14.217 INFO:teuthology.orchestra.run.smithi122.stderr: 90.8% -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-volume.log.gz 2024-04-16T06:03:14.223 INFO:teuthology.orchestra.run.smithi122.stderr: 96.7% -- replaced with /var/log/ceph/ceph-client.1.log.gz 2024-04-16T06:03:14.257 INFO:teuthology.orchestra.run.smithi122.stderr: 89.8% -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mgr.smithi122.sgokkq.log.gz 2024-04-16T06:03:14.281 INFO:teuthology.orchestra.run.smithi027.stderr: 93.9% -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi027.hkhpuq.log.gz 2024-04-16T06:03:14.291 INFO:teuthology.orchestra.run.smithi027.stderr: 94.1% -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mds.cephfs.smithi027.ejoytk.log.gz 2024-04-16T06:03:14.526 INFO:teuthology.orchestra.run.smithi122.stderr: 92.6% -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mon.smithi122.log.gz 2024-04-16T06:03:14.671 INFO:teuthology.orchestra.run.smithi027.stderr: 88.5% -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mgr.smithi027.qgzypi.log.gz 2024-04-16T06:03:14.783 INFO:teuthology.orchestra.run.smithi122.stderr: 93.3% -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-osd.4.log.gz 2024-04-16T06:03:14.818 INFO:teuthology.orchestra.run.smithi122.stderr: 93.7% -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-osd.5.log.gz 2024-04-16T06:03:14.887 INFO:teuthology.orchestra.run.smithi027.stderr: 93.3% -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-osd.2.log.gz 2024-04-16T06:03:14.893 INFO:teuthology.orchestra.run.smithi027.stderr: 93.4% -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-osd.0.log.gz 2024-04-16T06:03:14.930 INFO:teuthology.orchestra.run.smithi027.stderr: 93.4% -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-osd.1.log.gz 2024-04-16T06:03:14.946 INFO:teuthology.orchestra.run.smithi122.stderr: 93.5% -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-osd.3.log.gz 2024-04-16T06:03:14.950 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-04-16T06:03:14.950 INFO:teuthology.orchestra.run.smithi122.stderr:real 0m0.813s 2024-04-16T06:03:14.950 INFO:teuthology.orchestra.run.smithi122.stderr:user 0m2.290s 2024-04-16T06:03:14.950 INFO:teuthology.orchestra.run.smithi122.stderr:sys 0m0.313s 2024-04-16T06:03:15.300 INFO:teuthology.orchestra.run.smithi027.stderr: 90.4% -- replaced with /var/log/ceph/5e3cca06-fbb2-11ee-bc8f-c7b262605968/ceph-mon.smithi027.log.gz 2024-04-16T06:03:15.303 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-04-16T06:03:15.303 INFO:teuthology.orchestra.run.smithi027.stderr:real 0m1.186s 2024-04-16T06:03:15.304 INFO:teuthology.orchestra.run.smithi027.stderr:user 0m3.835s 2024-04-16T06:03:15.304 INFO:teuthology.orchestra.run.smithi027.stderr:sys 0m0.396s 2024-04-16T06:03:15.305 INFO:tasks.cephadm:Archiving logs... 2024-04-16T06:03:15.305 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-04-16_05:08:59-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657906/remote/smithi027/log 2024-04-16T06:03:15.306 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T06:03:15.886 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/log/ceph to /home/teuthworker/archive/vshankar-2024-04-16_05:08:59-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657906/remote/smithi122/log 2024-04-16T06:03:15.887 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T06:03:16.320 INFO:tasks.cephadm:Removing cluster... 2024-04-16T06:03:16.320 DEBUG:teuthology.orchestra.run.smithi027:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 --force 2024-04-16T06:03:17.411 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5e3cca06-fbb2-11ee-bc8f-c7b262605968 --force 2024-04-16T06:03:18.399 INFO:tasks.cephadm:Removing cephadm ... 2024-04-16T06:03:18.400 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-16T06:03:18.461 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-16T06:03:18.516 INFO:tasks.cephadm:Teardown complete 2024-04-16T06:03:18.517 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-16T06:03:18.530 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-16T06:03:18.531 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-16T06:03:18.531 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T06:03:18.539 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T06:03:18.695 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-16T06:03:18.706 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-16T06:03:18.706 DEBUG:teuthology.orchestra.run.smithi027:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T06:03:18.714 DEBUG:teuthology.orchestra.run.smithi122:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T06:03:18.765 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-04-16T06:03:18.768 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-04-16T06:03:18.775 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T06:03:18.776 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-04-16T06:03:18.776 INFO:teuthology.orchestra.run.smithi027.stdout:^* hv01.front.sepia.ceph.com 3 7 377 116 +1689us[+1686us] +/- 54ms 2024-04-16T06:03:18.776 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 50 +504us[ +504us] +/- 52ms 2024-04-16T06:03:18.776 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 53 -3368us[-3368us] +/- 67ms 2024-04-16T06:03:18.776 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T06:03:18.778 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T06:03:18.778 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-04-16T06:03:18.778 INFO:teuthology.orchestra.run.smithi122.stdout:^* hv01.front.sepia.ceph.com 3 6 377 52 +1820us[+1683us] +/- 54ms 2024-04-16T06:03:18.778 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 56 +569us[ +432us] +/- 52ms 2024-04-16T06:03:18.778 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 45 -3242us[-3242us] +/- 67ms 2024-04-16T06:03:18.778 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T06:03:18.779 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-16T06:03:18.792 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-16T06:03:18.793 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-16T06:03:18.808 DEBUG:teuthology.orchestra.run.smithi027:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T06:03:19.080 DEBUG:teuthology.orchestra.run.smithi122:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T06:03:19.314 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T06:03:19.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T06:03:19.420 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T06:03:19.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T06:03:19.525 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-16T06:03:19.537 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-16T06:03:19.566 INFO:teuthology.task.internal:Duration was 2572.187858 seconds 2024-04-16T06:03:19.567 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-16T06:03:19.580 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-16T06:03:19.580 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T06:03:19.588 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T06:03:19.737 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T06:03:19.742 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T06:03:20.161 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-16T06:03:20.161 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi027.front.sepia.ceph.com 2024-04-16T06:03:20.161 DEBUG:teuthology.orchestra.run.smithi027:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T06:03:20.220 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi122.front.sepia.ceph.com 2024-04-16T06:03:20.220 DEBUG:teuthology.orchestra.run.smithi122:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T06:03:20.310 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-16T06:03:20.311 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T06:03:20.318 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T06:03:20.414 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-16T06:03:20.414 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T06:03:20.419 DEBUG:teuthology.orchestra.run.smithi122:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T06:03:21.191 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-16T06:03:21.207 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-16T06:03:21.207 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T06:03:21.307 DEBUG:teuthology.orchestra.run.smithi122:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T06:03:21.407 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-16T06:03:21.420 DEBUG:teuthology.orchestra.run.smithi027:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T06:03:21.428 DEBUG:teuthology.orchestra.run.smithi122:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T06:03:21.536 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = core 2024-04-16T06:03:21.543 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = core 2024-04-16T06:03:21.620 DEBUG:teuthology.orchestra.run.smithi027:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T06:03:21.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T06:03:21.674 DEBUG:teuthology.orchestra.run.smithi122:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T06:03:21.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T06:03:21.728 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-16T06:03:21.741 INFO:teuthology.task.internal:Transferring archived files... 2024-04-16T06:03:21.742 DEBUG:teuthology.misc:Transferring archived files from smithi027:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-04-16_05:08:59-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657906/remote/smithi027 2024-04-16T06:03:21.742 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T06:03:21.877 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/vshankar-2024-04-16_05:08:59-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657906/remote/smithi122 2024-04-16T06:03:21.877 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T06:03:21.984 INFO:teuthology.task.internal:Removing archive directory... 2024-04-16T06:03:21.984 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T06:03:21.992 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T06:03:22.046 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-16T06:03:22.060 INFO:teuthology.task.internal:Not uploading archives. 2024-04-16T06:03:22.060 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-16T06:03:22.073 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-16T06:03:22.073 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T06:03:22.081 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T06:03:22.133 INFO:teuthology.orchestra.run.smithi027.stdout: 393249 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 16 06:03 /home/ubuntu/cephtest 2024-04-16T06:03:22.133 INFO:teuthology.orchestra.run.smithi027.stdout: 657186 4 d--------- 2 ubuntu ubuntu 4096 Apr 16 05:40 /home/ubuntu/cephtest/mnt.0 2024-04-16T06:03:22.133 INFO:teuthology.orchestra.run.smithi027.stderr:find: ‘/home/ubuntu/cephtest/mnt.0’: Permission denied 2024-04-16T06:03:22.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T06:03:22.136 INFO:teuthology.orchestra.run.smithi027.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-16T06:03:22.136 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa/tasks/ceph_fuse.py", line 183, in task mount.umount_wait() File "/home/teuthworker/src/github.com_vshankar_ceph_96e30c031480d623ee32b6336cfd0a901a764a51/qa/tasks/cephfs/fuse_mount.py", line 403, in umount_wait run.wait([self.fuse_daemon], timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-16T06:03:22.137 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-16T06:03:22.150 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-16T06:03:22.164 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2024-04-16T06:03:22.165 INFO:teuthology.nuke:Checking targets against current locks 2024-04-16T06:03:22.168 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi122.stderr:unique: 702, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 119483 2024-04-16T06:03:22.189 DEBUG:teuthology.nuke:shortname: smithi027 2024-04-16T06:03:22.190 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T06:03:22.210 DEBUG:teuthology.nuke:shortname: smithi122 2024-04-16T06:03:22.210 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T06:03:22.215 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-16_05:08:59-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657906', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 05:10:01.021081', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAUKOJJtb1jCnhSbBLTrDEMvl7nkN2zXTZYZ0tHH/3dXhQSwBrKySq1C+3BTRBW6QVgU06p2SdGdxkGp4SPmO50='} 2024-04-16T06:03:22.229 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-16_05:08:59-fs:upgrade:mds_upgrade_sequence-wip-vshankar-testing-20240412.042930-testing-default-smithi/7657906', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 05:10:01.022669', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFCcWLoCaHgr/ZCbWUO1jcAcLvGuaJdjPflJ4wydBj+pSal71/S2LrmF3xIHN+SVzZLGynxLMV2zqpzIVfr5bHI='} 2024-04-16T06:03:22.260 INFO:teuthology.orchestra.console.smithi027:Power off 2024-04-16T06:03:22.260 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T06:03:22.274 INFO:teuthology.orchestra.console.smithi122:Power off 2024-04-16T06:03:22.274 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T06:03:22.285 DEBUG:teuthology.orchestra.console.smithi027:power off output: Chassis Power Control: Down/Off 2024-04-16T06:03:22.285 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T06:03:22.297 DEBUG:teuthology.orchestra.console.smithi122:power off output: Chassis Power Control: Down/Off 2024-04-16T06:03:22.297 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T06:03:22.308 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-04-16T06:03:22.318 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2024-04-16T06:03:26.309 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T06:03:26.318 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T06:03:26.419 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2024-04-16T06:03:26.426 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2024-04-16T06:03:30.424 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T06:03:30.432 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T06:03:30.534 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is off 2024-04-16T06:03:30.541 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is off 2024-04-16T06:03:30.636 INFO:teuthology.orchestra.console.smithi027:Power off completed 2024-04-16T06:03:30.642 INFO:teuthology.orchestra.console.smithi122:Power off completed 2024-04-16T06:03:30.747 INFO:teuthology.run:Summary data: description: fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 2572.1878576278687 failure_reason: reached maximum tries (51) after waiting for 300 seconds flavor: default host.a-kernel-sha1: 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 host.b-kernel-sha1: 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 owner: scheduled_vshankar@teuthology status: fail success: false 2024-04-16T06:03:30.747 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T06:03:30.828 INFO:teuthology.run:FAIL